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1.  INTRODUCTION 


A.  Purpose  of  this  Document 

A  quality  assurance  mechanism  is  a  set  of  administrative 
procedures  designed  to  enhance  the  quality  of  services  de- 
livered by  either  organizational  or  individual  service 
providers  by  causing  or  encouraging  these  providers  to  come 
into  increasingly  greater  compliance  with  a  set  of  pre- 
established  standards.     In  the  case  at  hand,  the  standards 
define  desirable  characteristics  of  programs  and  services 

for  persons  with  developmental  disabilities.     A  model  set 
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of  such  standards  appears  in  a  companion  report.  The 
purpose  of  this  document  is  to  describe  a  set  of  administra- 
tive procedures  for  implementing  the  model  standards   (or  any 
set  of  standards  similar  in  nature) .     Together,  the  two 
reports  provide  guidance  for  a  state  to  establish  a  mechanism 
for  assuring  the  quality  of  its  developmental  disabilities 
service  system. 

B.  A  Synopsis  of  the  Model  Standards 

The  model  standards  are  of  two  types :  product  and  out- 
come standards,  which  focus  on  the  effectiveness  of  programs 
and  services  to  increase  and/or  maintain  the  competence  and/ 
or  well-being  of  persons  with  developmental  disabilities; 

1  "Model  Standards  To  Assess  the  Quality  of  Services  and  Programs 
for  Persons  with  Developmental  Disabilities"  revised  Januarv 
25,  1978. 


input  and  process  standards ,  which  focus  on  the  manner  in 
which  service  providers  carry  out  certain  functions  related 
to  case  finding  and  referral,   individual  program  planning 
and  case  management,  and  administrative  support,  and  how  they 
act  to  normalize  the  living  environment  and  to  protect  the 
rights  of  persons  with  developmental  disabilities. 

Two  research  and  analysis  activities  led  to  the  formula- 
tion of  the  set  of  model  standards.     The  product  and  outcome 
standards  on  the  one  hand,  derive  from  a  review  of  quality 
assurance  and  evaluation  literature  in  health,  education  and 
rehabilitation;  an  analysis  of  the  state-of-the-art  of  be- 
havioral assessment  for  persons  with  developmental  disabil- 
ities; and  a  review  of  the  literature  addressing  the  concept 
of  normalization.     The  input  and  process  standards,  on  the 
other  hand,  derive  from  a  careful  review  and  evaluation 
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of  previously  existing  standards.     An  earlier  project  report, 
describes  the  results  of  this  evaluation  by  recognized  ex- 

A* 

perts  in  the  field  of  developmental  disabilities.     The  method 
used  to  build  upon  these  findings  allowed  for  the  incorpora- 
tion of  the  best  features  of  existing  input  and  process 
standards,  the  adaptation  of  those  existing  standards  judged 
valid  but  unreliable  for  quality  assurance,  and  the  creation 
of  new  standards  where  none  existed. 

2  "Draft  Report:     Review  and  Evaluation  of  Current  Standards 
and  Quality  Assurance  Mechanisms"  July  29,  1977. 
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The  fact  that  the  model  standards  are  thus  freer  from 
the  deficiencies   (vagueness  of  wording,   lack  of  measurabil- 
ity  and  disproportionate  concern  with  structural  criteria) 
than  previously  existing  standards  is  one  regard  in  which 
they  are  model.     The  recommended  standards  are  also  model 
in  that  they  represent  ideals  towards  which  the  developmental 
disabilities  service  system  should  aspire,  rather  than 
minimum  standards  immediately  attainable  by  most  service 
providers . 

Certain  key  assumptions  regarding  activities  which 
should  have  taken  place  in  a  state  prior  to  an  attempt  to 
implement  the  model  standards  are  also  worthy  of  mention: 

1.  The  state  shall  have  designated  a  single  state 
agency   (the  QA  agency)   to  be  responsible  for 
implementing  the  standards  for  assuring  the  quality 
of  services  and  programs  to  persons  with  develop- 
mental disabilities;  except  in  states  small  in 
population  or  having  limited  financial  resources, 

it  is  recommended  that  the  QA  agency  operate  through 
substate  Regional  Coordinating  Agencies   (RCAs) ; 

2.  the  state  shall  have  developed  a  list  of  aoproved 
behavioral  assessment  instruments  for  ascertaining 
developmental  status  and  monitoring  developmental 
change  among  persons  with  developmental  disabilities; 
in  addition,   the  state  shall  have  specified  for  which 
subpopulations  each  instrument  is  to  be  used; 

3.  the  state  shall  have  developed  a  procedure  which 
uses  behavioral  assessment  findings  to  determine 
the  "developmental  level"  of  each  person  with  de- 
velopmental disabilities;  an  individual's  "develop- 
mental level"  shall  represent  a  gross  index  of 
his/her  level  of  functioning  measured  across  several 
behavior  and  skill  areas; 
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4.  the  state  shall  have  prescribed  a  standard  format 
for  Individual  Habilitation  Plans  so  that  key  IHP 
entries  -  assessment  findings,  goals,  objectives, 
needed  services,  service  plans  -  are  expressed 
using  a  standard  vocabulary  or  are  selected  from 
standardized  inventories  or  check  lists; 

5.  the  state  shall  have  established  a  client-centered 
information  system  which  shall  include  for  each 
person  with  developmental  disabilities  a  summary 
of  those  elements  of  his  IHP  essential  to  the 
quality  assurance  process;  except  in  states 
small  in  population  or  having  limited  financial 
resources,   it  is  likely  that  the  client-centered 
information  system  will  be  computerized. 

C.       An  Overview  of  the  Quality  Assurance  Process 

The  quality  assurance  process  divides  into  two  major 

phases :     the  design  phase  and  the  implementation  phase.  The 

design  phase  includes  those  activities  which  the  quality 

assurance  agency   (QA  agency)  must  complete  before  any  quality 

assessments  are  undertaken  in  any  given  review  cycle.  The 

implementation  phase  includes  those  tasks  which  the  QA  agency 

performs  each  time  it  carries  out  a  quality  assessment  (these 

tasks  are  thus  repeated  several  times  during  any  given  review 

cycle,  once  for  each  provider  reviewed). 

The  design  phase  includes  four  major  tasks: 

Task  1  -  Select  Criteria  and  Standards  and  Establish 
Decision  Rules  for  Reaching  Summary  Judgment 

During  this  task,  the  QA  agency  selects  the 
criteria  and  standards  it  will  use  to  assess  the 
quality  of  services  and  programs,   then  establishes 
the  relative  importance  of  each  standard  in  the  over- 
all determination  of  quality.     The  agency  accomplishes 
the  latter   (a)   by  assigning  a  priority  weight  to  each 
individual  standard;  and   (b)   by  establishing  decision 
rules  for  reaching  summary  judgments   (positive,  con- 
ditional or  negative)   as  to  the  quality  of  services 
being  delivered,  which  rules  reflect  the  provider's 
performance  standard-by-standard . 
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Task  2  -  Establish  Procedures  for  Implementation  Phase 


During  this  task,  the  QA  agency  establishes  the  pro- 
cedures to  be  followed  each  time  it  carries  out  a  quality 
assessment  of  a  particular  service  or  program.     In  documen 
ting  these  procedures,   the  QA  agency  identifies  the 
responsibility  of  all  actors  involved  and  assures  that 
the  quality  assessments  are  carried  out  in  a  consistent 
manner. 

Task  3  -  Establish  Qualifications  and  Develop  Training 
Programs  for  Quality  Assessors 

During  this  task,  the  QA  agency  takes  steps  to 
assure  that  persons  who  serve  as  quality  assessors  are 
suitably  qualified  and  adequately  trained. 

Taks  4  -  Select  and  Schedule  Service  Providers  to  be 
Reviewed  in  Upcoming  Review  Cycle 

During  this  task,  the  QA  agency  selects  the  pro- 
viders to  be  reviewed  in  the  upcoming  review  cycle,  es- 
tablishes a  sequence  for  the  reviews  and  sets  a  time 
table  for  their  completion. 

The  implementation  phase  involves  seven  major  tasks: 
Task  5  -  Initiate  Provider  Assessment 


During  this  task,  the  QA  agency  assigns  staff 
persons  to  the  impending  quality  assessment,  notifies 
the  provider  and  other  interested  parties,  and  establishes 
a  schedule  for  the  data  assembly  and  data  analysis  tasks 
which  follow. 

Task  6  -  Assemble  Data  Describing  Provider  Performance 

During  this  task,   the  QA  agency  assembles  all  the 
data  it  needs  to  assess  provider  performance  with  respect 
to  the  standards.     Selected  data  items  will  already  be 
available  to  the  QA  agency  or  to  substate  regional 
coordinating  agencies    (RCAs) .     Additional  data  will  be 
forwarded  by  the  provider  in  advance  of  a  site  visit 
by  the  on-site  quality  assessors.     The  site  visit  which 
follows  will  involve  examination  of  client  and  adminis- 
trative records,   interviews  with  members  of  the  provider 
staff  and  a  tour  of  the  provider  facilities. 
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Task  7  -  Analyze  Data  Describing  Provider  Performance 
and  Document  Findings 

During  this  task,  the  QA  agency  analyzes  the 
assembled  data  to  determine  the  provider's  performance 
(and  thus  level  of  compliance)  with  respect  to  each 
individual  standard.     The  task  culminates  with  the 
preparation  of  a  report  summarizing  the  QA  agency 
findings  standard-by-standard. 

Task  8  -  Reach  Summary  Judgment;  As  Necessary,  Identify 
Conditions  to  be  Corrected  and/or  Impose  Sanctions 

During  this  task,   the  QA  agency  uses  the  decision 
rules  developed  in  Task  1  together  with  the  findings 
arrived  at  in  Task  7  to  reach  a  summary  judgment  as  to 
the  quality  of  services  rendered  by  the  provider  being 
assessed.     If  a  conditional  judgment  is  reached,  the 
QA  agency  then  documents  any  conditions  to  be  corrected 
and  establishes  a  time  frame  within  which  such  correct- 
ions should  be  made.     If  a  negative  judgment  is  reached, 
the  QA  agency  not  only  identifies  critical  deficiencies, 
but  also  determines  or  recommends  sanctions  to  be  imposed 
upon  the  provider. 

Task  9  -  Communicate  Summary  Judgment  to  Provider 

During  this  task,   the  QA  agency  forwards  a  written 
report  which  summarizes  the  agency  findings,  informs 
the  provider  of  the  summary  judgment  reached,  enumerates 
any  corrections  to  be  made  by  the  provider,   and  identi- 
fies any  sanctions  to  be  imposed.     A  follow-up  meeting 
takes  place  between  the  QA  agency  staff  and  provider 
representatives . 

Task  10  -  Inform  Public  of  Summary  Judgment 

During  this  task,  the  QA  agency  informs  the  public 
of  the  summary  judgment  reached  in  accordance  with  pro- 
cedures established  in  Task  2. 

Task  11  --  Hear  Appeals  to  Summary  Judgment 

During  this  task,  the  QA  agency  will  consider  any 
appeals  to  the  summary  judgment  in  accordance  with 
procedures  established  in  Task  2  governing  who  may 
appeal,  grounds  for  appeals,   selection  of  an  appeals 
panel  and  documentation  of  the  resulting  judgment. 
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A  detailed  description  of  each  of  these  tasks  is  presented 
in  Chapter  2  of  this  report. 

D.     A  Framework  for  Self -Evaluation  of  Quality  Assurance 
Activities 

It  is  important  that  the  QA  agency  periodically  evaluate 
whether  or  not  its  quality  assurance  activities  are  having 
any  measurable  influence  upon  the  quality  of  programs  and  •• 
services  to  persons  with  developmental  disabilities.  For 
example,   the  QA  agency  should  attempt  to  determine  whether 
implementation  of  its  quality  assurance  mechanism  has  resulted 
in  improving  the  performance  of  programs  and  services,  as 
reflected  in  measurable  improvement  in  the  competence  or 
well-being  of  persons  with  developmental  disabilities,  and 
in  increased  provider  compliance  whith  input  and  process 
standards.     It  is  also  important  that  the  OA  agency  period- 
ically evaluate  whether  or  not  its  quality  assurance  process 
can  be  carried  out  in  a  more  efficient  or  more  equitable 
manner . 

It  is  therefore  recommended  that  the  QA  agency  also 
establish  procedures  for  the  self-evaluation  of  its  quality 
assurance  activities.     This  evaluation  should  focus  on  the 
following  factors: 

•     the  appropriateness  of  the  criteria  and  standards 
used  by  the  QA  agency  to  assess  the  quality  of 
services; 
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•  the  effectiveness  of  the  quality  assurance 
mechanism,  measured  primarily  in  terms  of  the 
change  over  time  in  the  proportion  of  providers 
who  meet  or  exceed  prescribed  levels  of 
performance; 

•  the  efficiency  of  the  quality  assurance  mechanism, 
measured  primarily  in  terms  of  the  manpower,  time 
and  financial  expenditures  associated  with  its 
execution; 

•  the  equity  of  the  quality  assurance  mechanism, 
in  particular  the  assurance  of  due  process,  im- 
partiality,  and  the  absence  of  any  infringement 
upon  the  rights  of  persons  with  developmental 
disabilities  or  service  providers. 

Analyses  to  be  carried  out  by  the  QA  agency  in  order  to  eval- 
uate its  own  activities  in  terms  of  these  factors  are  pre- 
sented in  Chapter  3. 
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2.      A  DETAILED  DESCRIPTION  OF  THE  QUALITY  ASSURANCE  PROCESS 


A.       The  Design  Phase 

The  design  phase  includes  those  activities  which  the  QA 
agency  must  complete  before  any  quality  assessments  are  under- 
taken in  any  given  review  cycle.     The  design  phase  includes 
four  major  tasks. 

Task  1  -  Select  Criteria  and  Standards  and  Establish 
Decision  Rules  for  Reaching  Summary  Judgment 

Task  2  -  Establish  Procedures  for  Implementation  Phase 

Task  3  -  Establish  Qualifications  and  Develop  Training 
~~~~~    Programs  for  Quality  Assessors 

Task  4  -  Select  and  Schedule  Service  Providers  to  be 
Reviewed  in  Upcoming  Review  Cycle 

These  tasks,  and  the  subtasks  within  each,  are  described 

in  detail  below. 

TASK  1  -  SELECT  CRITERIA  AND  STANDARDS  AND  ESTABLISH  DECISION 
RULES  FOR  REACHING  SUMMARY  JUDGMENT  ~~  

A  criterion  is  an  evaluative  scale,  measure  or  index.  A 
standard  is  the  desired  or  prescribed  level  of  performance  with 
respect  to  a  criterion.     Criteria  can  therefore  be  said  to 
structure  questions  which  should  be  asked  about  a  program  or 
service,  while  standards  specify  acceptable  responses  to  these 
questions.     Criteria  and  standards  thus  define  and  operationalze 
the  concept  of  "quality  services  for  persons  with  developmen- 
tal disabilities." 
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During  this  task,  the  QA  agency  selects  the  criteria  and 
standards  it  will  use  to  assess  the  quality  of  services  and 
programs,   then  establishes  the  relative  imoortance  of  each 
standard  in  the  overall  determination  of  quality.     The  agency 
accomplishes  the  latter   (a)  by  assigning  a  priority  weight 
to  each  individual  standard;  and   (b)   by  establishing  decision 
rules  for  reaching  summary  judgments   (positive,  conditional 
or  negative)   as  to  the  quality  of  services  being  delivered, 
which  rules  reflect  the  provider's  performance  standard-by- 
standard  . 

Five  subtasks  are  identified.     These  subtasks  are  listed 
below  and  are  discussed  in  detail  in  the  sections  which  follow 

1.1  Select  criteria  and  standards 

1.2  Assign  each  standard  to  a  priority  class 

1.3  Establish  levels  of  compliance  for  each  standard 

1.4  Establish  decision  rules  for  reaching  a  summary 
judgment 

1.5  Disseminate  standards 

Subtask  1.1  -  Select  Criteria  and  Standards 

The  criteria  and  standards  selected  by  the  QA  agency 

in  this  subtask  will  serve  as  the  basis  for  all  quality  assess 

3 

ments.     The  companion  report     to  this  model  quality  assurance 
mechanism  presents  a  comprehensive  set  of  model  standards 
which  could  be  adopted  by  a  QA  agency.     An  agency  should, 

3  "Model  Standards  to  Assess  the  Quality..."  Op.. cit. 
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however,  review  other  potential  sources  of  criteria  and 
standards,  including: 

•  standards  used  by  other  states  or  by  accreditation 
agencies  and  organizations ,  such  as  the  Accredita- 
tion Council  for  Services  for  Mentally  Retarded  and 
Other  Developmen tally  Disabled  Persons^ ; 

•  federal  and  state  legislation  and  regulations 
applicable  to  programs  and  services  for  persons 
with  developmental  disabilities; 

•  goals  and  objectives  found  in  state  developmental 
disabilities  plans  and  related  plans; 

•  statements  of  problems  and  issues  raised  by  the 
public,  government  agencies  and  legislators,  and 
private  organizations; 

•  self-evaluation  tools  for  service  providers  described 
in  the  literature  or  currently  in  use; 

•  articles  or  publications  which  identify  character- 
istics of  service  providers  contributing  to  or  im- 
pairing quality. 

The  QA  agency  should  also  consider  those  standards  that  will 
be  used  by  other  agencies  and  organizations,  such  as  those 
agencies  and  organizations  involved  in  granting  licenses  and 
those  involved  in  fire  and  safety  inspections.     The  QA  agency 
can  then  take  steps  to  assure  that  its  standards  are  consis- 
tent with  the  standards  of  these  other  agencies  and  organi- 
zations.    Moreover,  the  QA  agency  can  incorporate  among  its 
own  standards  the  requirement  that  service  providers  conform 


4  "Standards  for  Services  for  Developmental ly  Pisabled  Individ- 
uals   (A  revision  and  merger  of  the  Standards  for  Community 
Agencies  and  Standards  for  Residential  Facilities),"  Draft 
3,  September  20,  1977. 


11 


with  the  applicable  standards  established  by  these  other 
agencies,  thus  avoiding  duplicate  investigations. 

The  identification  of  potential  standards  by  QA  agency 
staff  members  is  followed  by  activities  which  address  two 
concerns:    (a)  that  the  criteria  and  standards  are  "good" 
indicators  of  quality  -  especially  that  each  criterion  is 
valid  and  reliable  and  the  associated  standards  appropriately 
stringent;    (b)   that  the  criteria  and  standards  are  comprehen- 
sive -  that  is,  that  the  set  of  criteria  and  standards  adequate 
ly  address  all  the  goals  of  the  developmental  disabilities  ser- 
vice system.' 

The  method  used  to  evaluate  previously  existing  standards 
during  the  development  of  the  model  input  and  process  standards 
is  described  in  an  earlier  report.^     A  OA  agency  might  choose  t 
assemble  a  small  team  of  outside  experts  and  use  this  method 
to  evaluate  any  standards  it  is  considering  for  adoption,  in- 
cluding any  standards  currently  in  force  in  its  state.  An 
external  evaluation  has  the  advantage  of  avoiding  any  possible 

appearance  of  conflict  of  interest  as  well  as  increasing  the 
credibility  of  the  evaluation  findings.     The  group  of  experts 
should  be  balanced  to  represent  the  interests  of  providers, 
advocates,  researchers  and  administrators. 


5     "Draft  Report:     Review  and  Evaluation..."  Op.  Cit. 
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Having  arrived  at  a  set  of  criteria  and  standards  which 
it  judges  to  be  "good"  indicators  of  quality,   the  QA  agency 
should  next  examine  this  set  to  determine  whether  or  not  the 
goals  of  the  developmental  disabilities  system  are  adequately 
addressed  by  the  standards.     These  goals  include: 

•  developmental  growth  -  that  each  individual  receive 
services,  in  accordance  with  an  individualized 
habilitation  plan   (IHP) ,  which  are  designed  to 
increase  or  maintain  the  individual's  physical, 
communicative,   social,   affective,  and  cognitive 
development,   as  manifested  in  increased  skill 
levels  and  achievement  of  adaptive  behaviors; 

•  normalization  -  that  persons  with  developmental 
disabilities  should  experience  patterns  of  daily 
living  as  close  as  possible  to  the  patterns  ex- 
perienced by  the  mainstream  of  society,  in  en- 
vironments which  likewise  are  as  culturally 
normative  as  possible; 

•  protection  of  individual  rights  -  that  persons  with 
developmental  disabilities  should  enjoy  basic  human 
and  civil  rights  and  protection  from  exploitation, 
neglect,  and  abuse. 

If  the  set  of  candidate  standards  does  not  reflect  the 
concerns  discussed  above,   the  QA  agency  staff  should  develop 
additional  standards,  or  adapt  existing  ones,  until  the  set  is 
comprehensive.     These  new  standards  should  be  evaluated  in- 
dividually and  the  set  reviewed  again  to  make  certain  it  is  now 
comprehensive . 

Some  standards  may  be  applicable  to  all  provider  types, 
regardless  of  the  services,  settings  and  client  groups  in- 
volved.    Other  standards  may  apply  only  to  particular  services, 
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settings  or  client  groups,  and  thus  only  to  selected  providers. 
The  statement  of  each  standard  should  thus  make  clear  to  which 
provider  type(s)   it  applies. 

Subtask  1.2  -  Assign  Each  Standard  to  a  Priority  Class 

Although  all  standards  used  for  quality  assessment  should 
have  previously  been  judged  "good"  indicators  of  quality,  it 
is  clear  that  not  all  standards  are  of  equal  importance. 
There  are  several  methods  for  assigning  priority  weights  to 
individual  standards  in  order  that  different  standards  have 
relatively  more  or  less  impact  on  the  overall  determination 
of  quality. 

Alternatives  include   Ca)   clustering  standards  into  two 
or  three  priority  classes   (high/low,  high/medium/ low)  with 
all  standards  in  each  class  assumed  equal  in  importance;  (b) 
placing  standards  into  a  priority  ranking  from  most  important 
to  least  important   (perhaps  using  a  simple  paired  comparison 
survey  method) ;    (c)   assigning  a  mathematical  weight  to  each 
standard  so  that  the  relative  importance  of  one  standard  versus 
any  other  standard  can  be  expressed  as  a  numerical  ratio. 

The  simplest  alternative  is  recommended  here  -  namely, 
that  each  input  and  process  standard  be  assigned  to  one  of 
three  priority  classes,  with  a  fourth  class  set  aside  for  all 
outcome  standards.     An  input  or  process  standard  should  be 
assigned  to  the  highest  priority  class  for  standards  of  this 
type   (class  1)   if  full  compliance  with  the  standard  is  judged 
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essential  to  quality.     An  input  or  process  standard  judged  to 
have  a  significant  impact  on  qualitv,  but  for  which  less  than 
full  compliance  is  acceptable,   should  be  assigned  to  class  2. 
All  remaining  input  and  process  standards  -  those  likely  to 
have  been  included  for  educational  purposes  -  should  be  assigned 
to  class  3.     Because  their  focus  is  so  different,  outcome 
standards  should  be  segregated  from  input  and  process  standards. 

An  illustrative  assignment  of  priorities  to  the  nine 
model  input  and  process  standards  related  to  Individual  Habil- 
itation  Plans  is  shown  in  Exhibit  I. 

Subtask  1.3  -  Establish  Levels  of  Compliance  for  Each  Standard 

Certain  standards  will  admit  only  two  levels  of  compliance: 

full  or  none.     An  example  of  such  a  standard  is  model  standard 

10.2     community  residences  and  residential  centers 

shall  conduct  at  least  one  building  evacuation 
drill  each  year  during  normal  sleeping  hours 

A  particular  community  residence  or  residential  center  is  either 

in  full  compliance  with  this  standard   (if  there  was  a  nighttime 

drill  during  the  past  year)   or  is  in  non-compliance   (if  there 

was  no  nighttime  drill) . 

Consider,  on  the  other  hand,  the  following  input  and 

process  standards,  which  also  appear  in  the  model  set: 

3.1     an  Individual  Habilitation  Plan   (IHP)   shall  be 
completed  for  each  person  with  developmental 
disabilities  within  45  days  of  the  completion 
of  his/her  initial  comprehensive  behavior  assess- 
ment; 
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5.4     each  individual  or  agency  which  provides  to  a 

person  with  developmental  disabilities  any  service 
identified  in  the  IHP  as  a  needed  service  shall 
maintain  a  record  of  service  delivery  which  shall 
include  at  least: 

-  the  frequency,  volume  and  nature  of  the  services 
provided; 

-  the  results  of  any  formal  assessments  conducted 
by  the  service  provider   (whether  before,  during 
or  after  the  period  during  which  services  were 
delivered) ; 

-  recommendations  for  future  service  needs; 

7.13     the  number  of  persons  sleeping  in  any  bedroom  shall 
not  exceed  four. 

Notwithstanding  that  full  compliance  with  each  of  these 

standards  is  the  ideal  condition,  it  is  clearly  advantageous 

in  these  cases  to  be  able  to  distinguish  the  degree  to  which 

a  provider  is    (is  not)   in  compliance  with  the  standard.  For 

example,  the  QA  agency  will  probably  want  to  distinguish 

between: 

•  a  provider  who  completes  IHPs  for  nearly  all  of 
its  clients  within  45  days  and  a  provider  who 
completes  IHPs  for  very  few  of  its  clients  within 
45  days; 

•  a  provider  who  keeps  records  of  service  delivery 
which  very  nearly  satisfy  the  content  requirements 
of  standard  5.4  and  a  provider  whose  records  show 
only  what  persons  were  receiving  services  at  any 
point  in  time; 

•  a  residential  center  where  some  bedrooms  are  designed 
for  six  or  eight  persons   (especially  children)  from 

a  residential  center  where  scores  of  persons  sleep  in 
large  barracks-like  rooms. 
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It  is  thus  recommended  that  the  QA  agency  define  up  to 

four  levels  of  compliance  for  each  standard: 

full  compliance,  which  indicates  that  the  provider 
meets  the  standard  in  all  respects; 

substantial  compliance ,  which  indicates  that  the 
provider  nearly  meets  the  standard  as  stated; 

minimal  compliance,  which  indicates  that  the  provider 
meets  only  the  minimum  performance  level  acceptable 
to  the  QA  agency  with  respect  to  the  standard; 

non-compliance ,  which  indicates  that  the  provider 
fails  to  meet  even  the  minimum  acceptable  performance 
level. 

Illustrative  levels  of  compliance  for  the  standards  in  section 
3  of  the  model  input  and  process  standards  appear  in  Exhibit 
II. 

Subtask  1.4  -  Establish  Decision  Rules  for  Reaching  a  Summary 
Judgment 

The  quality  assessment  of  the  services  of  a  given  provide 
will  result  in  a  positive,  conditional  or  negative  summary 
judgment.     A  positive  judgment  indicates  that  the  quality 
of  the  services  being  provided  is  acceptable  to  the  QA  agency. 
No  further  action  will  be  required  on  the  part  of  either  the 
provider  or  the  agency.     In  rendering  a  conditional  judgment, 
the  QA  agency  also  endorses  the  continued  provision  of  ser- 
vices by  the  provider,  but  in  this  case  the  endorsement  will 
be  contingent  upon  the  provider's  willingness  to  adhere  to 
conditions  of  correction  established  by  the  QA  agency. 
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In  rendering  a  negative  summary  judgment,  on  the  other 
hand,  the  QA  agency  rules  that  it  cannot  endorse  the  continued 
provision  of  services  by  the  provider  because  even  a  minimally 
acceptable  level  of  quality  is  not  being  achieved.  Negative 
judgments  are  thus  often  accompanied  by  the  imposition  of 
sanctions,  which  might  include  denial  of  reimbursement  for 
services  rendered,   contract  termination,  reduction  or  elimina- 
tion of  program  funding,   denial  of  licensure  or  certification, 
or  the  imposition  of  fines  and  penalities. 

What  is  necessary  at  this  step  in  the  design  phase  is 
that  the  QA  agency  develop  a  set  of  explicit  decision  rules 
for  determining  whether  it  will  render  a  positive,  a  conditional 
or  a  negative  summary  judgment.     The  decision  rules  should 
take  into  consideration  the  degree  to  which  the  provider  is 
in  compliance  with  all  applicable  standards,  with  due  regard 
for  the  priority  class  to  which  each  standard  was  assigned. 
Explicit  decision  rules  will  assure  that  summary  judgments 
are  reached  in  a  consistent  and  equitable  manner.     An  illustra- 
tive set  of  decision  rules,   suitable  for  use  if  a  OA  agency  were 
to  establish  a  structure  allowing  for  four  classes  of  standards 
and  up  to  four  levels  of  compliance  with  respect  to  each  stand- 
ard, appears  in  Exhibit  III. 
Subtask  1.5  -  Disseminate  Standards 

Before  any  assessments  of  quality  take  place,  the  OA 
agency  should  prepare  a  document  which  lists  the  selected 
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standards  and  the  types  of  providers  to  which  they  aoply, 
identifies  the  priority  class  to  which  each  standard  has 
been  assigned,   identifies  the  levels  of  compliance  established 
for  each  standard,   and  sets  forth  the  decision  rules  which 
will  be  used  in  making  summary  judgments.     This  document 
should  be  distributed  without  charge  to  each  individual  or 
organization  which  provides  services  to  persons  with  develop- 
mental disabilities  within  the  state  and  to  all  Regional 
Coordinating  Agencies    (RCAs) .     The  document  should  also  be 
distributed  at  no  charge,  or  at  cost,  to  other  interested 
parties . 

TASK  2  -   ESTABLISH  PROCEDURES   FOR  IMPLEMENTATION  PHASE 

During  this  task,  the  QA  agency  establishes  the  pro- 
cedures to  be  followed  each  time  it  carries  out  a  quality 
assessment  of  a  particular  service  or  program.     In  document- 
ing these  procedures,  the  QA  agency  identifies  the  respon- 
sibility of  all  actors  involved  and  assures  that  the  quality 
assessments  are  carried  out  in  a  consistent  manner. 
Four  subtasks  are  identified: 

2.1  Establish  mechanisms  for  protecting  client  rights 
during  quality  assurance  activities 

2.2  Establish  sequence  of  quality  assessment  tasks 

2.3  Establish  procedural  guidelines 

2.4  Disseminate  procedural  guidelines 


24 


Subtask  2.1  -  Establish  Mechanisms  for  Protecting  Client  Rights 
During  Quality  Assurance  Activities 

Quality  assurance  activities  should  be  designed  so  that 
all  personally  identifiable  data  are  assembled  and  disseminat- 
ed with  full  protection  of  individuals'   rights  to  privacy. 
The  QA  agency  should  thus  establish  written  procedures  to 
protect  client  rights,   requiring,  at  a  minimum, 

•  that  informed  consent  be  obtained  from  an 
individual    (or  his/her  parent  or  legal  guardian) 
whenever  personally  identifiable  data  are  ob- 
tained by  the  quality  assessors ,  whether 
directly  from  the  individual,   from  another 
respondent  regarding  the  individual,  or  from 
service  system  data  files; 

•  that  client  names  are  replaced  by  coded  iden- 
tifiers whenever  personally  identifiable  data 
are  entered  into  quality  assurance  data  files; 

•  that  all  client-specific  data  extracted  from 
quality  assurance  data  files  be  reported  in 
aggregate  form  only,  the  use  of  names  or  coded 
identifiers  being  allowed  only  with  the  informed 
consent  of  the  individuals  concerned. 

Subtask  2.2  -  Establish  Sequence  of  Quality  Assessment  Tasks 
In  this  subtask,   QA  agency  staff  establish  and  document 
the  sequence  of  tasks  that  will  together  constitute  an  assess- 
ment of  the  quality  of  the  services  provided  by  a  particular 
provider.     A  recommended  sequence  of  seven  tasks,  comprising 
the  implementation  phase  of  the  model  quality  assurance 
mechanism,  was  presented  in  Chapter  1.     These  tasks  will  be 
discussed  at  length  later  in  this  chapter. 
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Subtask  2.3  -  Establish  Procedural  Guidelines 

The  QA  agency  must  next  develop  a  set  of  procedural  guide' 
lines  which  describe  the  manner  in  which  each  step  in  the 
quality  assessment  process  is  to  be  carried  out  and  which 
identify  the  responsibility  of  all  persons  involved.  The 
preparation  of  procedural  guidelines  increases  the  likelihood 
that  quality  assessments  will  be  carried  out  in  a  uniform 
manner,  a  matter  of  legitimate  concern  to  providers. 

Later  in  this  chapter,  a  series  of  subtasks  is  defined 
for  each  task  in  the  implementation  phase  of  the  model  quality 
assurance  mechanism.     These  subtasks  represent  a  framework 
for  the  development  of  a  set  of  procedural  guidelines.  A 
QA  agency  seeking  to  implement  the  model  mechanism  can  use 
the  narrative  description  of  each  subtask  as  a  starting  point 
for  the  development  of  procedural  guidelines.     It  is  suggested 
that  particular  attention  be  given  to  the  following  activities 

•  assemble  self-assessment  data   (subtask  6.3) 
©     conduct  site  visit   (subtask  6.5) 

•  verify  assembled  data   (subtask  7.1) 
©     document  findings    (subtask  7.5) 

•  document  conditions  to  be  corrected   (subtask  8.2(a)) 

•  determine  sanction (s)   to  be  imposed   (subtask  8.3(b)) 

•  hold  appeals  hearing   (subtask  11.4) 
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Subtask  2.4  -  Disseminate  Procedural  Guidelines 

A  document  should  be  prepared  and  distributed  by  the  QA 
agency  which  describes  the  procedural  guidelines  to  be 
followed  during  quality  assessments.     Like  the  standards 
document,   the  procedures  document  should  be  sent,  without 
charge,  to  all  providers  and  RCAs  in  the  state,  and  should 
also  be  made  available  to  other  interested  parties  at  no 
charge  or  cost. 

TASK  3  -  ESTABLISH  QUALIFICATIONS  AND  DEVELOP  TRAINING  PRO- 
GRAMS FOR  QUALITY  ASSESSORS 

However  well  designed  a  quality  assurance  mechanism  may 
be,  it  can  not  be  effectively  administered  without  adequately 
trained  assessors.     Quality  assessors  will  examine  client 
and  administrative  records,   interview  members  of  the  provider 
staff  and  tour  the  provider  facilities,  both  to  assemble  new 
data  critical  to  the  quality  assessment  and  to  verify  data 
obtained  from  other  sources. 

The  people  who  make  up  the  assessment  team  are  of  two 
types:     staff  of  the  QA  agency  and  consultants  to  the  agency 
who  participate  only  in  on-site  assessments.     This  task  de- 
scribes activities  undertaken  in  the  design  phase  to  assure 
that  the  persons  who  serve  as  aualitv  assessors  are  suitably 
qualified  and  adequately  trained. 
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Three  subtasks  are  identified: 

3.1  Establish  minimum  qualifications  for  qualitv 
assessors 

3.2  Develop  and  implement  training  program 

3.3  Develop  and  implement  program  for  maintaining 
the  competence  of  quality  assessors 

Subtask  3.1  -  Establish  Minimum  Qualifications  for  Quality 
Assessors 

The  QA  agency  should  establish  minimum  qualifications  of 
education  and  experience  for  those  persons  who  will  be  employed 
as  quality  assessors.     These  qualifications  should  be  suffi- 
ciently high  to  satisfy  providers  and  others  that  the  assess- 
ors are  capable  of  making  reliable  judgments  about  providers' 
compliance  with  the  standards.     The  qualifications  should  be 
documented  in  detail  with  any  acceptable  equivalent  qualifica- 
tions clearly  stated. 

Subtask  3.2  -  Develop  and  Implement  Training  Program 

To  further  increase  the  capabilities  of  those  persons  who 
will  conduct  assessments,  the  QA  agency  should  develop  and 
implement  a  training  program  designed  to  thoroughly  familiarize 
the  quality  assessors  with  both  the  standards  and  the  procedures 
to  be  used  during  the  assessments.     Workshops  and  seminars  are 
useful  means  of  providing  an  initial  orientation.     There  should 
follow  a  training  period  for  any  candidate  quality  assessor 
during  which  time  the  candidate  accompanies  one  or  more 
assessment  teams  as  an  observer.     Subsequently,  an  assessor 
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may  be  assigned  to  a  team  as  a  junior  member,  working  under 

the  direction  of  someone  with  more  experience.     During  both 

the  training  period  and  the  period  of  initial  service,  the 

new  assessor  should  be  tested  for  inter-rater  reliability  against 

more  experienced  team  members . 

Subtask  3.3  -  Develop  and  Implement  Program  for  Maintaining 
the  Competence  of  Quality  Assessors 

The  accuracy  and  credibility  of  the  quality  assurance 
process  depends,   to  a  great  extent,  on  the  competence  of  the 
QA  agency  staff  and  consultants  who  conduct  assessments. 
For  this  reason,   it  is  important  that  the  QA  agency  adminis- 
ter a  continuing  education  program  for  all  assessors.  A 
workshop,  held  at  least  once  each  year,  should  be  attended 
by  all  assessors.     During  the  workshop,   inter-rater  reliability 
should  be  verified  through  case  studies,  and  any  standards 
or  procedures  frequently  challenged  by  providers  should  be 
carefully  examined  and  discussed.     The  workshop  also  provides 
an  opportunity  for  discussion  of  any  standards  or  procedures 
found  by  the  assessors  to  be  difficult  to  implement,  in- 
dicating that  adaptation  or  clarification  is  needed. 

TASK  4   -   SELECT  AND  SCHEDULE  SERVICE  PROVIDERS  TO  BE  REVIET'7ED 
IN  UPCOMING  REVIEW  CYCLE 

During  this  task,  the  QA  agency  selects  the  providers 

to  be  reviewed  in  the  upcoming  review  cycle,  establishes  a 

sequence  for  the  reviews  and  sets  a  time  table  for  their 

completion.     Providers  who  consistently  receive  high  quality 
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assessment  ratings  need  not  be  reviewed  by  the  QA  agency  during 
each  review  cycle.     Instead,  a  disproportionate  effort  should 
be  directed  towards  providers  known  to  be  poorer  performers 
on  the  basis  of  prior  assessments  and  to  providers  who  appear 
accountable  for  any  observed  undesirable  outcomes  experienced 
by  persons  with  developmental  disabilities. 

Another  factor  to  be  considered  in  scheduling  reviews 
during  a  single  future  review  cycle  is  the  desirability  of 
having  quality  assurance  personnel  available  to  meet  "crises" 
which  may  arise.     Serious  cases  of  abuse  or  other  violations 
of  the  rights  or  safety  of  persons  with  developmental  disabil- 
ities may  occur,   necessitating  immediate  review  of  one  or  more 
service  providers  by  the  QA  agency.     To  assure  its  capabilitv 
to  respond  to  such  situations,   the  QA  agency  may  elect  to  not 
completely  allocate  all  of  its  quality  assurance  personnel 
in  its  initial  schedule  of  reviews. 

The  selection  of  service  providers  and  the  scheduling  of 

reviews  involves  five  subtasks : 

4.1  Identify  service  providers 

4.2  Categorize  service  providers 

4.3  Establish  maximum  time  between  reviews  for  each 
type  of  provider 

4.4  Select  providers  to  be  reviewed  in  upcoming  cycle 

4.5  Establish  sequence  of  reviews  in  upcoming  cycle 
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Subtask  4.1  -  Identify  Service  Providers 


QA  agency  staff  should  identify  by  name  and  address 
each  provider  of  services  to  persons  with  developmental 
disabilities  in  the  state.     Each  RCA  in  the  state  should 
maintain  a  list  of  providers  within  its  area  and  these 
lists  should  be  assembled  by  the  QA  agency  into  a  state-wide 
inventory . 

Subtask  4.2  -  Categorize  Service  Providers 

The  inventory  of  service  providers  should  be  categorized 
by  the  QA  agency  staff.     At  least  three  primary  categories 
are  recommended: 

•  residential  center  -  a  residential  setting,  usually 
large  in  size,  where  the  capability  exists  to  pro- 
vide an  individual  with  developmental  disabilities 
the  full  range  of  developmental  and  generic  ser- 
vices he/she  requires; 

•  community  residence  -  a  home-like  residence  situated 
in  the  community  whose    (developmentally  disabled) 
residents  receive  support  and  supervision  from  live- 
in  staff  or  from  staff  persons  who  are  assigned  to 
the   residence  but  do  not  live  in; 

•  day  program  -  a  program  which  provides  services  of 
at  least  three  hours  continuous  duration  between  the 
hours  of  breakfast  and  dinner  at  a  site  away  from 

a  person's  usual  place  of  residence;  day  programs  are 
typically  provided  at  schools,  sheltered  workshops, 
day  care  activity  centers  and  rehabilitation 
facilities . 

A  fourth  category  for  other  non-residential  service  providers 
may  be  appropriate. 

States  may  also  find  it  desirable  to  develop  subcategories 
within  each  of  these  primary  categories.     Thus,  residential 
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centers  might  be  differentiated  by  size,   group  homes  might 
be  distinguished  from  sheltered  apartments  within  the  community 
residence  category,  and  day  programs  might  be  differentiated  by 
the  nature  of  the  services  they  provide. 

Subtask  4.3  -  Establish  Maximum  Time  Between  Reviews  for 
Each  Type  of  Provider 

While,  on  the  one  hand,   the  annual  review  of  all  providers 
may  not  be  cost-effective,  the  QA  agency  will  also  want  to 
assure  that  there  is  not  an  excessive  amount  of  time  between 
successive  reviews  of  any  given  provider.     The  OA  agency  should 
therefore  decide  during  the  design  phase  the  maximum  time  it 
will  allow  between  assessments  of  providers.     The  maximum  time 
will  likely  vary  from  one  provider  category  to  another. 
Subtask  4.4  -  Select  Providers  to  be  Reviewed  in  Upcoming  Cycle 

Various  approaches  can  be  used  in  selecting  service  pro- 
viders to  be  reviewed  in  a  forthcoming  review  cycle.  Among 
these  approaches  are  the  following: 

©     regularly  scheduled  reviews  -  each  service  provider 
Ts  reviewed  once  every  X  vears .     The  number  of 
years  between  reviews  may  be  different  for  different 
types  of  service  providers  as  discussed  in  subtask 
4 . 3  above; 

©     providers  selected  for  review  on  the  basis  of 
past  performance  -  particular  service  providers 
or  types  of  service  providers  known  to  be  poorer 
performers    (based  upon  past  quality  assessments) 
may  be  reviewed  more  frequently) ; 

•     providers  selected  for  review  on  the  basis  of  ob- 
served undesirable  outcomes  experienced  by  persons 
for  whose  competence  and  well-being  the  provider 
is  accountable ; 
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•     providers  selected  for  reviews  on  a  random  basis  - 
a  predetermined  proportion  of  the  service  providers 
is  selected  for  review  each  year- 
Selecting  providers  on  the  basis  of  past  performance  or 
observed  outcomes  will  generally  be  more  efficient  because  o: 
the  greater  likelihood  of  identifying  programs  and  services 
in  need  of  quality  improvement.     However,  the  random  selec- 
tion of  providers  for  review  has  an  added  benefit  in  that 
it  may  tend  to  keep  service  providers  more  "on  their  toes." 

The  selection  procedure  described  in  this  sub-task 
combines  all  four  selection  approaches  into  a  three-phase 
process  for  determining  which  service  providers  will  be  re- 
viewed in  the  forthcoming  cycle: 

©     first,  providers  who  must  be  reviewed  in  the  forth- 
coming cycle  are  identified  on  the  basis  of  factors 
which  include   (a)   the  time  elapsed  since  the  most 
recent  review   (of  each  provider) ;  and   (b)  the 
findings  of  all  reviews  in  the  immediately  pre- 
ceding review  cycle; 

©     second,   additional  providers  are  selected  for 
review  using  an  outcome-oriented  method  which 
attempts  to  identify  programs  or  service  providers 
most  accountable  for  observed  undesirable  outcomes 
experienced  by   (groups  of)   persons  with  developmen- 
tal disabilities; 

©     third,  additional  providers  are  selected  for  re- 
view on  a  random  basis. 

The  providers  who  must  be  reviewed  include  those  who 

are  due  for  a  review  because  the  maximum  allowable  time  has 

elapsed  since  their  last  review.     Also  in  this  class  are  pro 

viders  who  were  given  a  conditional  judgment  during  the  pre- 
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ceding  review  cycle  and  who  were  required  to  undertake 
corrective  actions  which  cannot  be  verified  except  by  a  re- 
assessment.    To  the  list  of  service  providers  who  must  be 
reviewed  the  QA  agency  may  also  add  those  service  providers 
about  whom  there  have  been  complaints   (from  persons  with 
developmental  disabilities,  their  families  or  representatives) 
or  reports    (from  other  reviewing  agencies  or  organizations) 
which  indicate  the  existence  of  conditions  serious  enough  to 
necessitate  a  full-scale  review  of  the  service  provider  by 
the  QA  agency. 

The  outcome-oriented  selection  of  additional  service  pro- 
viders should  begin  with  the  identification  of  those  groups 
of  persons  with  developmental  disabilities  who  have  exper- 
ienced less  than  satisfactory  outcomes  during  the  past  year. 
Less  than  satisfactory  outcomes  would  include  lack  of  develop- 
mental progress    (or  little  progress  where  much  was  expected) 
or  the  absence  of  positive  change  with  respect  to  selected 
normalization  indicators.     Such  data  should  be  extracted  from 
a  sample  of  individual  IHPs  or  from  client-centered  information 
systems  which  include  behavioral  assessment  data  as  well  as  in- 
formation reflecting  the  attainment  or  non-attainment  of 
individual  habilitation  objectives.     Statewide  quality-of-life 

surveys  may  also  identify  population  subgroups   (by  age,  develop- 
mental level,   type  of  residence  or  substate  region  of  residence) 
experiencing  extremely  unfavorable  degrees  of  overall  well- 
being. 
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In  either  case,  the  next  step  is  to  identify  those  ser- 
vice providers  who  can  apparently  be  held  accountable  for  any 
observed  undesirable  outcomes.     It  is  from  among  this  group 
that  the  secondary  selection  of  service  providers  to  be  in- 
cluded in  the  forthcoming  review  cycle  should  be  made.  This 
approach  will  assure  that  all  service  providers  are  considered 
in  the  outcome-based  selection,  not  just  those  providers  with 
records  of  poor  performance  in  past  quality  assessments. 

The  number  of  providers  chosen  in  this  second  phase  of 
the  selection  process  will  depend  upon  several  factors:  (a) 
the  number  of  mandatory  reviews  to  be  carried  out;    (b)  the 
proportion  of  QA  agency  manpower    (if  any)    set  aside  for 
randomly  scheduled  reviews;    (c)    the  proportion  of  QA  agency 
manpower   (if  any)   reserved  for  reviews  necessitated  by  "crisis" 
conditions.     In  any  case,  a  method  is  needed  to  put  into 
priority  order  those  service  providers  who  are  candidates  for 
quality  assurance  review  on  the  basis  of  observed  outcomes. 
Clearly,   it  is  also  necessary  that  the  QA  agency  have  accurate 
estimates  of  the  amount  of  time  and  the  level  of  effort  required 
to  complete  quality  assessments  for  each  provider  type. 

Once  it  has  determined  the  number  of  service  providers 
(if  any)   to  be  selected  for  review  on  a  random  basis,  the  QA 
agency  can  use  any  random  sampling  method  to  choose  from  among 
those  providers  not  selected  in  either  the  first  or  second  phase 
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of  the  selection  process.     Sampling  fractions  may  vary  from 
one  provider  category  to  another. 

Sub-task  4.5  -  Establish  Sequence  of  Reviews  in  Upcoming  Cycle 

In  this  final  step  in  the  design  phase,  the  QA  agency 
establishes  a  sequence  for  the  reviews  for  the  upcoming  cycle 
and  determines  a  timetable  for  their  completion.     The  fact 
that  the  QA  agency  schedules  reviews  over  the  entire  forth- 
coming review  cycle  does  not  imply  that  the  agency  should  make 
that  schedule  public  from  the  outset.     The  greater  the  extent 
of  prior  notice,   the  greater  the  likelihood  that  a  service  pro 
vider  might  attempt  to  distort  or  manipulate  data  to  enhance 
appearances  during  the  quality  assessment. 
B .       The  Implementation  Phase 

The  implementation  phase  includes  those  tasks  which  the 
QA  agency  performs  each  time  it  carries  out  a  quality  assess- 
ment.    These  tasks  are  thus  repeated  as  many  times  during  any 
review  cycle  as  there  are  providers  to  be  reviewed  during  that 
cycle.     It  is  assumed  that  all  tasks  in  the  design  phase  are 
completed  prior  to  the  initiation  of  any  quality  assessments 
in  any  given  review  cycle. 

The  implementation  phase  includes  seven  major  tasks: 

Task  5  -  Initiate  Provider  Assessment 

Task  6  -  Assemble  Data  Describing  Provider  Performance 

Task  7  -  Analyze  Data  Describing  Provider  Performance 
and  Document  Findings 
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Task  8  -  Reach  Summary  Judgment;  As  Necessary,  Identify 
Conditions  to  be  Corrected  and/or  Impose  Sanctions  ~ 

Task  9  -  Communicate  Summary  Judgment  to  Provider 

Task  10  -  Inform  Public  of  Summary  Judgment 

Task  11  -  Hear  Appeals  to  Summary  Judgment 

These  tasks,  and  the  subtasks  within  each,  are  described  in 

detail  below. 

TASK  5   -   INITIATE  PROVIDER  ASSESSMENT 

During  this  task,   the  QA  agency  assigns  staff  persons  to 
the  impending  quality  assessment,  notifies  the  provider  and 
other  interested  parties,  and  establishes  a  schedule  for  the 
data  assembly  and  data  analysis  tasks  which  follow. 

Five  subtasks  are  identified: 

5.1  Assign  staff 

5.2  Identify  applicable  standards 

5.3  Notify  provider 

5.4  Notify-  other  interested  parties 

5.5  Establish  schedule  for  data  collection  and  analysis 
Subtask  5.1  -  Assign  Staff 

The  Director  of  the  OA  agency  should  assign  particular 

staff  members  to  the  impending  assessment,   including  any 
consultants  who  will  take  part  in  the  on-site  data  collection 
task.     One  staff  member  should  be  given  the  responsibility 
for  the  assessment  on  behalf  of  the  QA  agency.     The  number  and 
type  of  other  team  members  will  vary  with  the  size  and  type 
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of  provider  of  serivces  being  assessed.     Only  those  who  satis- 
fy the  requirements  established  in  Task  3  of  the  design  phase 
should  be  included  in  the  assessment  team. 
Subtask  5.2  -  Identify  Applicable  Standards 

The  compendium  of  standards  selected  during  the  design 
phase  includes  some  standards  applicable  to  all  provider  types, 
regardless  of  the  services,  settings  and  client  groups  involved, 
and  some  standards  applicable  only  to  particular  services, 
settings  or  client  groups,   and  thus  only  to  selected  providers. 
Prior  to  undertaking  the  assessment  of  a  given  provider,  OA 
agency  staff  should  thus  identify  those  standards  applicable 
to  that  provider.     The  relevant  standards  can  then  be  assembled 
for  the  data  collection  and  analysis  tasks  which  follow. 
Subtask  5.3  -  Notify  Provider 

A  formal  written  notification  of  the  impending  assessment 
should  be  sent  to  the  provider  by  the  Director  of  the  QA 
agency.     This  notification  should  describe  why  the  provider 
was  selected  for  review  during  the  current  cycle,  should  out- 
line the  procedures  to  be  followed  during  the  quality  assess- 
ment,  and  should  identify  the  name  of  the  QA  agency  staff 

member  with  primary  responsibility  for  the  assessment. 

The  amount  of  advance  notice  given  to  a  provider  might 

vary  with  the  type  of  provider,  being  longer,   for  example,  for 

providers  serving  large  numbers  of  clients  or  providing  a 

wide  variety  of  services,  since  the  volume  and  complexity 

of  data  will  be  greater  in  such  instances. 
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Subtask  5.4  -  Notify  Other  Interested  Parties 

A  formal  written  notification  of  the  impending  assessment 
is  sent  by  the  Director  of  the  QA  agency  to  those  persons  and 
organizations  identified  as  other  parties  interested  in  the 
assessment  of  the  given  provider.     The  notification  should 
include  a  deadline  for  the  submission  of  comments  to  the 
QA  agency  by  other  interested  parties . 

Subtask  5.5  -  Establish  Schedule  for  Data  Collection  and  Analysis 

QA  agency  staff  assigned  to  the  assessment  should  establish 
a  schedule  for  the  data  collection  and  analysis  tasks  which 
follow.     The  provider  should  be  consulted  to  determine,  within 
reasonable  limits,  the  most  convenient  dates  for  the  on-site 
data  collection  task. 

TASK  6   -  ASSEMBLE  DATA  DESCRIBING  PROVIDER  PERFORMANCE 

During  this  task,  the  OA  agency  assembles  all  the  data 
it  needs  to  assess  provider  performance  with  respect  to  the 
standards.     Selected  data  items  will  already  be  available  to 
the  QA  agency  or  to  substate  regional  coordinating  agencies 
(RCAs).     Additional  data  will  be  forwarded  by  the  provider  in 
advance  of  a  site  visit  by  the  on-site  quality  assessors.  The 
site  visit  which  follows  will  involve  examination  of  client 
and  administrative  records,   interviews  with  members  of  the 
provider  staff  and  a  tour  of  the  provider  facilities. 

In  all  data  collection  activities  the  QA  agency  should  be 
guided  by  several  principles:      (a)   that  the  data  items  which 
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are  collected  are  as  accurate  and  timely  as  possible;  (b) 
that  the  personal  and  legal  rights  of  persons  with  develop- 
mental disabilities  are  not  violated;  and   (c)    that  the  routine 
activities  of  the  service  provider  being  assessed  are  disrupted 
as  little  as  possible. 

Task  6  includes  the  following  five  subtasks : 

6.1  Assemble  data  internal  to  the  QA  agency 

5.2  Assemble  data  from  RCAs 

6.3  Assemble  self-assessment  data  from  provider 

6.4  Assemble  data  from  other  interested  parties 

6.5  Conduct  site  visit 

Subtask  6.1  -  Assemble  Data  Internal  to  the  OA  Agency 

Some  of  the  data  items  needed  for  the  assessment  will  al- 
ready have  been  collected  by  the  OA  agency  for  other  purposes. 
Model  standard  11.4,   for  example,   requires  that  residential 
centers  and  community  residences  forward  to  the  QA  agency  re- 
ports of  all  inspections  by  state  or  local  building   (fire  and 
safety)   or  health  authorities.     It  might  also  be  the  case  that 
the  provider  has  furnished  special  reports  to  the  QA  agency, 
including  accounts  of  corrections  made  by  the  provider  in 
response  to  a  conditional  judgment  during  a  prior  assessment. 


40 


Subtask  6.2  -  Assemble  Data  from  RCAs 

The  functional  responsibilities  assigned  to  the  RCAs 
within  a  given  state  will  determine  the  extent  to  which  the 
QA  agency  can  rely  upon  the  RCAs  as  sources  of  data  for  quality 
assessments.     A  companion  report  °  recommends  several  key 
functions  for  the  RCAs:      (1)   each  RCA  is  to  maintain  a 
registry  of  all  persons  with  developmental  disabilities  who 
reside  within  the  geographic  area  served  by  the  RCA;  (2) 
each  RCA  is  to  maintain  the  comprehensive  unit  record  of  all 
persons  who  reside  outside  a  residential  center;    (3)   each  RCA 
is  responsible  for  responding  to  grievances  lodged  against 
service  providers  and  for  investigating  reported  cases  of 
negelct  or  abuse. 

The  first  and  second  of  these  functions  make  the  RCAs 
a  source  of  considerable  data  with  respect  to  client  progress, 
measured  in  terms  of  developmental  growth  as  well  as  normaliza- 
tion of  life  style.     The  third  function  enables  the  RCA  to 
assist  in  the  identification  of  providers  who  abridge  the 
rights  of  persons  with  developmental  disabilities.     An  obvious 
concern  is  that  the  RCAs  throughout  the  state  maintain  uniform 
data  systems  for  both  client-specific  and  provider-specific 
information. 


6  "Model  Standards  to  Assess  the  Quality..."  Op.  cit. 
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Subtask  6.3  -  Assemble  Self-Assessment  Data  from  Provider 

Prior  to  the  site  visit,  the  QA  agency  should  forward  to 
the  provider  copies  of  the  data  collection  forms  which  will  be 

used  by  the  on-site  quality  assessors.     In  the  letter  of 
transmittal,  the  QA  agency  should  also  request  that  the  pro- 
vider carry  out  a  partial  self-assessment  by  completing  all 
those  data  items  which  do  not  require  sampling  of  records 
or  interviews  with  staff  personnel. 

This  opportunity  for  self-assessment  is  useful  to  both 
the  provider  and  the  QA  agency.     By  enabling  the  provider  to 
identify  certain  deficiencies  in  advance  of  the  formal  quality 
assessment,   it  allows  the  provider  the  opportunity  to  correct 
minor  problems  in  advance  of  the  site  visit  and  to  begin  to 
design  plans  of  correction  for  more  substantial  deficiencies. 
At  the  same  time,   it  can  reduce  the  time  required  for  the  on- 
site  survey  because  it  enables  the  provider  to  assemble  in 
advance  any  supporting  evidence  the  on-site  assessors  will 
need  to  verify  responses  to  selected  data  items. 

The  completed  self-assessment  form  should  be  returned 
to  the  QA  agency  in  sufficient  time  to  allow  those  conducting 
the  site  visit  an  opportunity  to  review  the  provider  response 
several  days  prior  to  the  on-site  assessment. 
Subtask  6.4  -  Assemble  Data  from  Other  Interested  Parties 

The  deadline  for  the  submission  of  written  comments  by 
other  interested  parties  should  be  no  later  than  the  deadline 
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for  the  return  of  the  provider  self-assessment  survey  form. 
All  comments  received  should  be  reviewed  by  the  persons  who 
will  conduct  the  on-site  assessment  since  the  comments  may  in- 
fluence the  manner  in  which  the  on-site  assessors  choose  sample 
records  or  select  staff  persons  to  interview. 

Subtask  6.5  -  Conduct  Site  Visit 

The  final  step  in  the  data  assembly  task  requires  a 
team  of  QA  agency  staff  persons  and/or  outside  consultants 
to  visit  the  service  provider  site.     The  purpose  of  the  site 
visit  is  threefold:      (1)   to  assemble  data  by  sampling  client 
or  administrative  records  and  bv  interviewing  members  of  the 
provider  staff;    (2)   to  verify  data  furnished  by  the  provider 
in  the  aftermath  of  the  self-assessment  activity;    (3)   to  verify 
data  assembled  from  other  sources. 

The  size  of  the  site-visit  team  should  be  determined 
by  the  nature  and  size  of  the  provider  being  assessed.  The 
nature  of  the  provider  may  also  influence  the  desirable  mix 
of  qualifications  of  the  on-site  assessors.     The  QA  agency 
staff  person  with  primary  responsibility  for  the  assessment 
should  be  designated  as  site  visit  team  leader. 

The  first  site  visit  activity  should  be  a  meeting  between 
the  site  visit  team  and  the  Director   and  senior  staff  of  the 
provider.     The  purpose  of  this  meeting  is  to  review  the  site- 
visit  protocol  and  to  acquaint  the  quality  assessors  with  the 
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information  sources    (both  personal  and  physical)   they  will 

utilize.     Members  of  the  site  visit  team  should  then  separate, 

with  each  responsible  for  gathering  data  from  a  different 

source.     A  team  of  at  least  three  quality  assessors  is 

recommended:     one  to  assemble  data  from  client  and  administrative 

records,  one  to  interview  staff  personnel  and  one  to  conduct  a 

physical  tour  of  the  facilities. 

All  client-specific  data  for  persons  served  by  the  provider 

in  question  should  be  available  from  the  comprehensive  unit 

record  maintained  for  each  person  with  developmental  disabilities. 

7 

According  to  the  companion  report, '    the  comorehensive  unit 

record  should  contain  at  least  the  following  information; 

©     the  person's  name,  date  of  birth  and  place  of 
resicence 

e     the  results  of  all  comprehensive  behavioral 
assessments  within  the  last  five  years 

©     all  records  of  service  delivery  within  the 
last  five  years 

•  all  IHPs  developed  within  the  last  five  years 

©     annual  assessments  of  the  person's  need  for  a 

legal  guardian  or  a  guardian  of  his/her  property 

•  annual  assessments  of  the  person's  physical  and 
mental  health 

•  all  physician's  orders  within  the  last  two  vears 

•  all  pertinent  correspondence 


7  "Model  Standards  to  Assess  the  Quality..."  Op.  cit. 
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However,   except  in  the  case  of  a  residential  center,  the  com- 
prehensive unit  records  are  not  likely  to  be  at  the  provider 
site.     The  survey  team  will  thus  need  to  determine  in  advance 
if  any  on-site  records  will  provide  the  necessary  information, 
or  if  arrangements  must  be  made  to  draw  a  sample  of  comprehensive 
unit  records  in  advance  of  the  site  visit.     In  any  case,  the 
sample  should  be  large  enough  to  be  a  statistically  valid  rep- 
resentation of  the  universe  of  persons  served.  Furthermore, 
the  sample  should  be  stratified  by  age  and  developmental  level 
to  reflect  the  mix  of  persons  served  with  respect  to  these 
variables  . 

Interviews  with  staff  members  of  the  provider  being 
assessed  are  required  to  gather  some  data  items.     Staff  inter- 
views may  be  useful  in  determining  the  extent  to  which  pro- 
viders adhere  to  written  policies  and  procedures  with  respect 
to  the  proper  use  of  physical  restraints,   the  proper  use  of 
psychotrophic  medication,  and  the  prohibition  of  corporal 
punishment,   verbal  and  physical  abuse,   seclusion  and  denial 
of  food  as  punishment. 

The  site  visit  also  must  include  a  tour  of  the  facility. 
Many  of  the  standards  addressing  the  normalization  of  the 
living  environment  -  such  as  the  type,  size,   furnishing  and 
other  characteristics  of  rooms  -  can  only  be  assessed  through 
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direct  observation.     Finally"/  at  the  end  of  the  site  visit, 
the  team  members  should  again  meet  with  the  Director  and 
senior  staff  of  the  provider  to  briefly  review  the  data  which 
has  been  assembled.     Such  an  exit  interview  provides  an 
opportunity  for  on-site  correction  of  any  data  which  might 
have  been  inaccurately  recorded. 

TASK   7   -   ANALYZE  DATA  DESCRIBING  PROVIDER  PERFORMANCE  AND 
DOCUMENT  FINDINGS 

During  this  task,   the  QA  agency  analyzes  the  assembled 
data  to  determine  the  provider's  performance   (and  thus  level 
of  compliance)  with  respect  to  each  individual  standard.  The 
task  culminates  with  the  preparation  of  a  report  summarizing 
the  QA  agency  findings  standard-by-standard. 

Five  subtasks  are  identified: 

7.1  Verify  assembled  data 

7.2  Gather  necessary  comparison  data 

7.3  Determine  degree  of  compliance  with  respect  to 
each  standard 

7.4  Determine  degree  of  compliance  with  respect  to 
standards  in  each  priority  class 

7.5  Prepare  summary  report  of  quality  assessment 
findings 

Subtask  7.1  -  Verify  Assembled  Data 

Activities  in  this  subtask  assure  that  provider  performance 
is  determined  on  the  basis  of  accurate  data.     QA  agency  staff 
should  verify  the  assembled  data  through  several  means.  Cer- 
tain data  items  will  have  been  gathered  from  more  than  one 
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source.     Any  discrepancy  between  these  sources  may  indicate 
inaccuracies  and  the  data  should  be  re-checked.     If  the  provider 
has  undergone  a  previous  assessment,  the  results  of  that  assess- 
ment should  be  compared  with  those  of  current  assessment  to 
determine  whether  any  observed  changes  in  results  are  reasonable. 
Unreasonable  changes  may  indicate  inaccurate  data.     Any  apparent 
errors  and  inconsistencies  identified  in  this  way  should  be  re- 
solved by  the  OA  agency  staff  before  proceeding  to  the  tasks 
which  follow. 

Subtask  7.2  -  Gather  Necessary  Comparison  Data 

Certain  outcome  standards  assess  provider  performance  by 
comparing  attainments  of  persons  with  developmental  disabili- 
ties served  by  that  provider  with  those  of  a  suitable  com- 
parison group.     QA  agency  staff  members  should  gather  the 
necessary  data  from  a  representative  comparison  group  to  de- 
termine provider  performance  by  means  of  such  comparisons. 

Subtask  7.3  -  Determine  Degree  of  Compliance  with  Resoect  to 
Each  Standard 

In  this  subtask  the  QA  agency  staff  analyses  provider 
performance  with  respect  to  each  applicable  standard,  using 
comparison  data  where  necessary.     The  end  result  will  be  a 
determination,   for  each  standard,   that  the  provider  is  (a) 
in  full  compliance;    (b)    in  substantial  compliance;    (c)  in 
minimal  compliance;  or   (d)   in  non-compliance.     For  certain 
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standards,   the  level  of  compliance  will  be  expressed  in  terms 

of  percentages   (e.g.,   the  oercentage  of  persons  in  the  sample 

grouo) ,   averages    (e.g.,   the  average  size  of  multi-resident 

bedrooms),   or  frequencies  of  occurrence    (e.g.,   the  number  of 

times  information  was  released  from  client  unit  records  without 

prior  approval).     In  these  cases,   simDle  analyses  may  be 

necessary  in  order  to  determine  the  level  of  convoliance. 

Subtask  7.4  -  Determine  Degree  of  Compliance  with  Respect  to 
Standards  in  Each  Priority  Class 

The  recommended  decision  rules  for  reaching  a  summary 

judgment  are  based  upon  the  degree  to  which  a  provider  is  in 

full,   substantial,  minimal  or  non-compliance  with  all  the 

standards  in  each  priority  class.     The  aggregation  of  the 

standard-by-standard  determinations  of  compliance  (from 

subtask  7.3)    into  determinations  of  compliance  by  priority 

class  will  thus  permit  direct  application  of  the  recommended 

decision  rules.     If  the  decision  rules  are  expressed  in  a 

different  form,   a  different  set  of  calculations  will  be 

needed  in  this  subtask. 

Subtask  7.5  -  Prepare  Summary  Report  of  Quality  Assessment 
Findings 

At  this  point,   the  QA  agency  staff  person  with  primary 
responsibility  for  the  assessment  prepares  a  report  summarizing 
the  quality  assessment  findings.     The  report  should  contain  a 
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brief  narrative  description  of  the  data  collection  and  analysis 
procedures  used,   followed  by  a  presentation  of  the  standard-by- 
standard  compliance  ratings.     All  determinations  of  less  than 
full  compliance  should  be  accompanied  by  a  brief  explanation 
of  the  reasons  for  the  lesser  rating.     A  summary  of  the  com- 
pliance ratings  by  priority  class  should  follow  the  standard- 
by-standard  findings. 

TASK   8   -   REACH   SUMMARY  JUDGMENT;    AS  NECESSARY,    IDENTIFY  CON- 
DITIONS TO  BE  CORRECTED  AND/OR  IMPOSE  SANCTIONS 

During  this  task,   the  QA  agency  uses  the  decision  rules 

developed  in  task  1  together  with  the  findings  arrived  at  in 

task  7  to  reach  a  summary  judgment  as  to  the  qualitv  of  ser- 

ivces  rendered  by  the  provider  being  assessed.     If  a 
conditional  judgment  is  reached,   the  QA  agency  then  documents 
any  conditions  to  be  corrected  and  establishes  a  time  frame 
within  which  such  corrections  should  be  made.     If  a  negative 
judgment  is  reached,   the  OA  agency  not  only  identifies  critical 
deficiencies,   but  also  determines  or  recommends  sanctions 
to  be  imposed  upon  the  provider.     Sanctions  might  include 
denial  of  remibursement  for  services  rendered,  contract 
termination,   reduction  or  elimination  of  program  funding, 
denial  of  licensure  or  certification,  or  the  imposition  of 
fines  and  penalties. 
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The  subtasks  which  comprise  this  task  vary  with  the 


judgment  made  and  are  discussed  below  as  alternatives: 


8.1 


Document  positive  judgment 


8.2 (a) 


Document  conditional  judgment 


8.2 (b) 


Determine  timetable  for  corrections  and  establish 
mechanisms  for  notification  of  corrections  made 


8.  3 (a) 


Document  negative  judgment  and  identify  critical 
deficiencies 


8.3(b)   Determine  sanctions  to  be  imposed 
An  agency  must  carry  out  subtask  8.1,   or  subtasks  8.2(a)  and 
8.2(b),  or  subtasks   8.3(a)    and  8.3(b). 
Subtask  8.1  -  Document  Positive  Judgment 

If  application  of  the  decision  rules  established  in  sub- 
task  1.4  leads  to  the  conclusion  that  the  provider  is  providing 
services  of  adequate    (or  exemplary)   quality,   the  QA  agency 
documents  this  judgment  by  means  of  a  brief  written  state- 
ment.    The  written  statement  will  subsequently  be  forwarded 
to  the  provider  along  with  the  summary  report  of  the  quality 
assessment  findings.     By  identifying  all  standards  with  which 
the  provider  was  in  less  than  full  compliance,   the  summary 
report  serves  an  important  educational  function,   the  positive 
judgment  notwithstanding. 

Subtask  8.2(a)    -  Document  Conditional  Judgment  and  Identify 
Conditions  to  be  Corrected 

If  application  of  the  decision  rules  leads  to  a  condition- 
al judgment,  the  QA  agency  should  not  only  document  which  as- 
pect of  the  decision  rules  and  findings  led  to  the  conditional 
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judgment,  but  should  also  identify  all  corrections  which  must 
be  made  in  order  for  the  QA  agency  to  endorse  the  continued 
provision  of  services  by  the  provider.     If,   for  example,  a 
conditional  judgment  was  reached  because  the  provider  was  in 
less  than  full  compliance  with  certain  standards  judged  essen- 
tial to  quality,   the  QA  agency  should  not  only  prepare  a 
written  statement  to  that  effect,  but  should  also  prepare  a 
supplemental  report  which  identifies  the  desired  levels  of 
performance  or  desired  conditions  not  currently  met  with 
respect  to  each  of  these  standards.     The  written  statement  of 
the  conditional  judgment  and  the  supplemental  report  document- 
ing the  conditions  to  be  corrected  will  subsequently  be  for- 
warded to  the  provider  along  with  the  summary  report  of  the 
quality  assessment  findings. 

Subtask  8.2(b)   -  Determine  Timetable  for  Corrections  and  Es- 
tablish Mechanism  for  Notification  of  Corrections  Made 

In  this  subtask,   the  QA  agency  staff  establishes  a  time- 
table for  the  correction  of  all  circumstances  which  caused 
the  QA  agency  to  render  a  conditional  summary  judgment. 
Deadlines  for  corrections  may  vary  according  to  the  importance 
to  clients,   complexity  or  cost  of  the  changes  to  be  made.  At 
the  same  time,   the  QA  agency  should  determine  a  suitable  means 
for  provider  to  notify  the  agency  that  necessary  corrections 
have  been  made.     In  some  cases,   a  letter  from  the  provider  or 
a  copy  of  an  inspection  report  from  another  agency  will 
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suffice.     In  other  cases,  either  a  follow-up  site  survey  or 

a  complete  re-assessment  might  be  necessary. 

Subtask  8.3(a)   -  Document  Negative  Judgment  and  Identify 
Critical  Deficiencies 

Application  of  the  decision  rules  leads  to  a  negative 
summary  judgment  when  provider  performance  fails  even  to 
satisfy  that  level  of  quality  which  the  QA  agency  deems 
minimally  acceptable  for  the  continued  provision  of  services. 
As  in  the  case  of  a  conditional  judgment,   the  OA  agency  staff 
should  carefully  document  the  reasons  for  the  negative  judg- 
ment by  preparing  a  supplemental  report  which  identifies 
standard-by-standard  all  disparities  between  provider  per- 
formance and  minimally  acceptable   levels  of  quality.     In  the 
case  of  an  appeal  by  the  provider,   the  supplemental  report  will 
serve  as  preliminary  evidence  in  support  of  the  negative 
summary  judgment  reached  by  the  QA  agency. 
Subtask  8.3(b)   -  Determine  Sanction (s)    to  be  Imnosed 

Because  a  negative  judgment  implies  that  the  quality  of 

services  is  not  even  minimally  acceptable,  negative  judgments 
are  often  accompanied  by  the  imposition  of  sanctions.     In  the 
design  phase  of  its  activities,   the  QA  agency  will  have  de- 
termined which  sanctions  might  be  imposed  in  the  case  of  neg- 
ative summary  judgments. 

A  frequent  sanction  is  the  denial  of  reimbursement  for 
services  rendered  to  identified  clients.     Unless  the  QA 
agency  itself  contracts  with  providers    (vendors) ,  the  only 
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manner  in  which  it  can  act  to  impose  sanctions  of  this  nature 
is  to  recommend  to  contracting  agencies  or  third-party  payors 
that  they  deny  reimbursement  to  the  provider  in  question. 

The  denial  of  program  funding  is  an  alternative  financial 
sanction.     Although  most  QA  agencies  will  not  have  the  authority 
to  deny  program  funding,   they  may  make  recommendations  to 
funding  agencies  based  on  the  results  of  the  quality  assurance 
process.     Since  many  service  providers  receive  substantial 
funding  support  from  federal,   state  and  local  government 
agencies,   the  loss  of  such  funds  may  have  a  major  impact. 

Some  QA  agencies  may  also  be  able  to  control  or  influence 
statewide  certification  or  licensure  programs.     Loss  of  certifi- 
cation may  subsequently  result  in  loss  of  reimbursement  or 
other  funding  support.     The  suspension  or  revocation  of  a 
license  to  operate  bears  with  it  an  unmatched  element  of 
finality . 

Other  sanctions  include  the  removal  of  accreditation  or 
the  imposition  of  fines.     Although  neither  is  likely  to  be  an 
action  which  can  be  directly  taken  by  the  OA  agency,   the  agency 
could  recommend  to    (professional)   accrediting  organizations 
that  they  undertake  a  reaudit  of  a  previously  accredited  pro- 
vider, while  the  imposition  of  fines  by  means  of  the  legal 
process  may  in  part  follow  from  the  agency's  quality  assessment 
findings . 
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TASK  9   -   COMMUNICATE   SUMMARY  JUDGMENT  TO  PROVIDER 

During  this  task,   the  QA  agency  forwards  a  written  report 
which  summarizes  the  agency  findings,   informs  the  provider  of 
the  summary  judgment  reached,   enumerates  any  corrections  to 
be  made  by  the  provider,  and  identifies  any  sanctions  to  be 
imposed.     A  follow-up  meeting  takes  place  between  the  QA  agency 
staff  and  provider  representatives.     The  two  subtasks  are 
therefore : 

9.1  Send  written  notification  of  summarv  judgment 
and  supporting  documentation  to  provider 

9.2  Meet  with  provider  staff 

Subtask  9.1  -  Send  Written  Notification  of  Summary  Judgment 
and  Supporting  Documentation  to  Provider 

The  purpose  of  this  subtask  is  threefold:      (1)    to  inform 
the  provider  of  the  summary  judgment  of  the  OA  agency  with  resnect 
to  the  quality  of  the  provider's  services;    (2)    to  document  the 
reasons  why  that  particular  judgment    (positive,   conditional  or 
negative)   was  reached;   and   (3)    to  identify  those  conditions 
which  prevented  a  positive   (or  conditional)    judgment  from 
being  rendered. 

In  all  cases,   a  written  statement  of  the  summary  judgment 
and  a  copy  of  the  Summary  Report  of  Quality  Assessment  Find- 
ings   (see  subtask  7.5)   is  sent  to  the  provider.     In  the  case 
of  a  conditional  judgment,  the  following  additional  material 
is  forwarded:      (a)   the  supplemental  report   (see  subtask  8.2(a)) 
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which  identifies  the  desired  levels  of  performance  or  desired 
conditions  not  currently  met;    (b)    a  timetable  for  corrections 
and  a  mechanism  for  notification  of  corrections  made    (see  sub- 
task  8.2(b));    (c)   a  citation  of  the  authority  under  which  the 
agency  is  empowered  to  impose  conditions  of  correction. 

In  the  case  of  a  negative  judgment,   the  additional 
documentation  includes:      (a)   the  supplemental  report   (see  sub- 
task  8.3(a))   which  identifies  all  disparities  between  provider 
performance  and  minimally  acceptable  levels  of  quality;  (b) 
a  precise  description  of  the  sanction (s)  which  the  QA  agency 
will  itself  impose  or  the   (punitive)   action (s)   it  will 
recommend  on  the  part  of  other  agencies  or  organizations 
(see  subtask  8.3(b));    (c)    a  citation  of  the  authority  under 
which  the  agency  is  empowered  to  impose  sanctions;    (d)  identi- 
fication of  the  earliest  date  upon  which  the  QA  agency  will 
entertain  a  request  from  the  provider  for  a  reassessment. 
The  establishment  of  such  a  date  assures  that  the  OA  agency 
is  not  burdened  by  requests  for  reassessment  until  there  is 
a  reasonable  chance  that  the  ncesssary  corrections  have  been 
completed . 

Subtask  9.2  -  Meet  with  Provider  Staff 

A  follow-up  meeting  between  QA  agency  staff  and  provider 
representatives  is  advisable  for  a  number  of  reasons.  First, 
it  encourages  a  cooperative  rather  than  an  adversary  relation- 
ship.    Second,   it  allows  an  opportunity  for  discussion  of  the 
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agency  judgment,   for  elaboration  of  any  unclear  aspects  of  the 
supporting  documentation,   and  for  clarification  of  any  next 
steps  to  be  undertaken  by  either  the  agency  or  the  provider. 
Third,   if  the  provider  is  considering  an  appeal  to  the 
summary  judgment,   the  appeal  procedures  can  be  explained  at 
this  time.     Finally,   it  allows  an  opportunity  for  negotiation 
between  the  agency  and  the  provider  regarding  either  the  changes 
to  be  made  and  the  timetable  for  such  changes  in  the  case  of  a 
conditional  judgment,  or  the  changes  to  be  made  and  the  earliest 
date  for  reassessment  in  the  case  of  a  negative  judgment.  Any 
compromises  reached  as  a  result  of  the  meeting  must,  however, 
be  documented  in  an  exchange  of  correspondence  between  the  OA 
agency  and  the  provider. 

TASK   10   -    INFORM  PUBLIC  OF  SUMMARY  JUDGMENT 

During  this  task,   the  QA  agency  informs  the  public  of 
the  summary  judgment  reached.     The  method  to  be  used  for  in- 
forming the  public  and  the  extent  of  any  additional  information 
provided  will  have  been  determined  in  task  2. 
TASK  11   -   HEAR  APPEALS   TO  SUMMARY  JUDGMENT 

It  is  recommended  that  two  levels  of  appeal  be  allowed. 
At  the  first  level  of  appeal   (the  one  of  interest  here) ,  the 
provider  or  another  interested  party  petitions  the  QA  agency 
to  change  its  summary  judgment,  either  because  the  accuracy 
of  certain  data  items  which  contributed  to  the  summary  judgment 
is  challenged,  or  because  it  is  alleged  that  certain  of  the 
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quality  assessment  procedures  were  improperly  executed.  The 
second  potential  level  of  appeal,   to  some  higher  administrative 
or  judicial  body,   is  outside  the  scope  of  the  quality  assurance 
mechanism  per  se. 

Who  may  appeal  the  judgment  and  on  what  grounds  appeals 
may  be  made  should  be  determined  by  the  OA  agency  in  task  2. 
At  the  same  time,   the  agency  should  also  establish  the  pro- 
cedures for  selecting  an  appeals  panel,   for  holding  an 
appeals  hearing  and  for  documenting  any  resulting  judgments. 
Carrying  out  the  appeals  process  at  this  first  level  of  appeal 
involves  six  subtasks  : 

11.1  Select  panel  for  appeals  hearing 

11.2  Select  date  for  appeals  hearing 

11.3  Inform  provider  and  other  interested  parties 
of  appeals  hearing 

11.4  Hold  appeals  hearing 

11.5  Document  appeals  findings 

11.6  Make  appeal  findings  public 
Subtask  11.1  -  Select  Panel  for  Appeals  Hearing 

In  this  subtask  the  QA  agency  selects  a  panel  to  hear 
the  appeal  which  seeks  a  change  in  judgment  by  the  agency. 
The  composition  of  such  a  panel  is  an  option  left  to  in- 
dividual states,  but  panel  members  should  probably  include 
senior  agency  staff  personnel  or  outside  consultants  who  did 
not  participate  in  the  initial  assessment  rather  than  those 
who  did. 
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Subtask  11.2  -  Select  Data  for  Appeals  Hearing 

The  date  of  the  appeals  hearing  should  be  established 

sufficiently  in  advance  of  the  hearing  to  allow  the  parties 

at  interest  enough  time  to  prepare  for  the  hearing,  and  to 

allow  those  selected  to  serve  on  the  panel  an  opportunity 

to  become  thoroughly  familiar  with  the  assessment  findings. 

Subtask  11.3  -  Inform  Provider  and  Other  Interested  Parties 
of  Appeals  Hearing 

As  soon  as  the  panel  has  been  selected  and  the  date 
set  for  the  hearing,  the  provider  and  other  interested  parties 
should  be  sent  written  notification  of  the  impending  appeals 
hearing.     This  notification  should  include  the  date,  time 
and  place  of  the  hearing,  who  has  made  the  appeal  and  the 
grounds  upon  which  the  appeal  is  being  made. 
Subtask  11.4  -  Hold  Appeals  Hearing 

Procedures  for  holding  the  appeals  hearing  will  be 
significantly  influenced  by  each  state's  administrative  codes. 
The  appeal  procedures  should  provide  an  opportunity  for  the 
QA  agency  staff  person  with  primary  responsibility  for  the 
assessment  to  offer  a  rebuttal  to  the  challenges  offered  by 
the  party  making  the  appeal. 
Subtask  11.5  -  Document  Appeal  Findings 

The  findings  of  the  appeal  panel  should  be  documented 
in  detail  by  the  QA  agency  staff.     The  arguments  for  and 
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against  a  change  in  judgment,  evidence  presented  by  both 
sides  and  the  reasons  for  the  appeal  decision  should  be 
present . 

Subtask  11.6  -   f-fake  Appeal  Findings  Public 

Finally,   the  Director  of  the  QA  agency  should  make  the 
appeals   findings  public  through  the  channels  used  to  inform 
the  public  of  earlier  agency  decisions.     If  the  party 
bringing  appeal  has  given  written  notification  to  the  QA 
agency  of  its  intention  to  seek  a  second  appeal  at  a  higher 
level,   this  fact  might  be  included  in  the  QA  agency  statement. 
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3.      EVALUATION  OF  THE  QUALITY  ASSURANCE  MECHANISM 


The  QA  agency  should  establish  procedures  for  self-evalua- 
tion of  its  quality  assurance  activities.     This  evaluation  should 


focus  on  the  following  factors: 


•     the  appropriateness  of  the  criteria  and  standards 
used  by  the  QA  agency  to  assess  the  quality  of 
services; 

e     the  effectiveness  of  the  quality  assurance 
mechanism,  measured  primarily  in  terms  of 
the  change  over  time  in  the  proportion  of 
providers  who  meet  or  exceed  prescribed 
levels  of  performance; 


®     the  efficiency  of  the  quality  assurance 

mechanism,  measured  primarily  in  terms  of  the 
manpower,   time  and  financial  expenditures 
associated  with  its  execution; 


©     the  equity  of  the  quality  assurance  mechanism 
which  focuses  primarily  on  the  assurance  of 
due  process,   impartiality,   and  the  absence  of 
any  infringement  upon  the  rights  of  persons 
with  developmental  disabilities  or  service 
providers . 

The  QA  agency  may  wish  to  carry  out  certain  self -evaluation 
activities  on  an  annual  basis.     Other  analyses  might  be 


carried  out  as  infrequently  as  once  every  three  years.  Re- 
gardless of  the  schedule  which  is  followed,    it  is  important 
that  the  QA  agency  record  information  needed  for  the  self- 
evaluation  during  the  assessment  of  individual  service 
providers . 

The  purpose  of  this  chapter  is  threefold: 

1)     to  identify  those  factors  to  be  considered  by  a 
QA  agency  in  evaluating  the  appropriateness  of 
standards  it  uses  to  assess  quality  and  the 
effectiveness,  efficiency  and  equity  of  its 
quality  assurance  procedures; 
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2) 


to  describe  analyses  to  be 
agency  to  evaluate  its  own 
of  these  factors; 


carried  out  by  the  QA 
activities  in  terms 


3)     to  describe  the  data  which  the  QA  agency  should 
collect  and  maintain  in  order  to  carry  out  these 
self-evaluation  analyses. 

Each  of  these  purposes  is  discussed  in  the  sections 

which  follow. 

A.     Appropriateness  of  the  Standards 

During  the  design  phase  of  the  quality  assurance 
mechanism  the  QA  agency  selected  a  set  of  criteria  and  stan- 
dards for  use  in  quality  assurance.     An  important  component  of 
the  self-evaluation  of  the  QA  agency  is  the  periodic  re-examin- 
ation of  the  selected  standards  to  determine  whether  or  not 
there  is  continued  reason  to  believe  that  each  is  a  "good" 
indicator  of  quality. 

The  appropriateness  of  the  criteria  and  standards  used  to 
assess  the  quality  of  services  can  be  measured  in  terms  of  the 
following  four  factors: 

1)  validity  -  the  validity  of  a  criterion 
(hence,   the  associated  standard)    is  measured 
by  the  strength  of  its  causal  relationship 
with  developmental  growth,   normalization  and/ 
or  the  protection  of  individual  rights; 

2)  reliability  -  the  reliability  of  a  criterion 
(hence,   the  associated  standard)   is  its  ability, 
under  different  circumstances  and  over  time,  to 
accurately  measure  a  specific  characteristic  of 
a  service  or  program; 
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3)  attainability /stringency  -  the  respective  concerns 
are,   on  the  one  hand,  whether  the  desired  or  pre- 
scribed level  of  performance    (the  standard)  is 
neither  so  high  as  to  be  unattainable  nor  so  low  as 
to  represent  less  than  a  minimally  acceptable  degree 
of  quality; 

4)  applicability  -  the  applicability  of  a  criterion 
(hence,   the  associated  standard)   is  the  extent  to 

which  corrective  action  to  change  the  phenomenon 
addressed  by  the  criterion    (standard)    lies  within 
the  jurisdiction  or  responsibility  of  the  provider 
of  the  services  to  which  the  standard  is  applicable. 

Once  a  sufficient  number  of  quality  assessments  has  been 
completed,   the  appropriateness  of  individual  criteria  and  stand- 
ards can  be  judged  by  comparative  analyses  of  provider  perform- 
ance against  each  standard.     Suggested  analyses  are  described 
in  the  following  sections. 
Validity 

If  one  or  several  input  and  process  standards  are  believed 
conducive  to  a  particular  positive  outcome,  validity  can  be 
judged  by  comparing  the  observed  outcomes  against  the  degree  of 
provider  compliance  with  the  input  and  process  standard (s)  in 
question.     If,   for  example,  a  group  of  related  input  and  process 
standards  were  initially  judged  to  be  predictive  of  developmen- 
tal growth,   then  individuals  served  by  providers  who  are  in 
greater  compliance  with  these  input  and  process  standards  should 
exhibit   (on  the  average)   greater  developmental  growth  than  in- 
dividuals served  by  providers  in  lesser  compliance  with  the  same 
set  of  input  and  process  standards. 
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Reliability 

One  measure  of  the  reliability  of  a  criterion  is  the 
extent  to  which  the  criterion,   if  repeatedly  used,  yields 
similar  results.     Data  gathered  from  the  same  service  pro- 
vider on  two  or  more  successive  quality  assessments  should 
therefore  be  compared,  and  unusually  large  changes  from  one 
assessment  to  the  next  should  be  investigated.     If  provider 
performance  with  respect  to  a  given  standard  is  found  to 
fluctuate  widely  without  apparent  explanation,   the  criterion 
itself  is  probably  unreliable,  perhaps  because  no  unit  of 
measure  is  specified,  making  provider  performance  largely  a 
matter  of  subjective  judgment. 

Data  accuracy  is  another  measure  of  the  reliability  of 
a  criterion   (standard) .     In  determining  provider  compliance 
with  each  standard,   allowance  should  be  made  for  a  category 
labelled  "determination  unable  to  be  made."     Reasons  why  it 
might  not  have  been  possible  to  make  a  determination  with  res- 
pect to  a  given  standard  include  the  following:      (a)     data  were 
unavailable;    (b)     there  was  a  data  input  error  by  the  quality 
assessment  team;    (c)     data  were  distorted  or  manipulated  by  the 
provider;    (d)     there  were  irreconciliable  differences  in  data 
gathered  from  two  or  more  information  sources;    (e)     the  sample 
of  individuals  or  records  used  to  determine  compliance  with  the 
standard  was  inappropriately  drawn.     The  QA  agency  should 
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maintain  historical  data  on  all  reasons  for  "determination 
unable  to  be  made"  and  should  use  these  data  to  identify 
those  standards  for  which  accurate  data  are  generally  not 
available . 

Attainability/Stringency 

The  QA  agency  should  also  develop  historical  data  des- 
cribing the  proportion  of  providers  in  full  compliance,  sub- 
stantial compliance,  minimal  compliance  and  non-compliance  with 
each  individual  standard.     Furthermore,   for  each  quantitative 
standard,   the  QA  agency  should  examine  by  provider  type  the 
minimum,  maximum  and  mean  level  of  provider  performance. 
Analyses  of  this  type  will  show  which  standards  are  most 
difficult  to  comply  with  and  which  are  easiest  to  comply  with. 
Subsequent  deliberations  may  lead  the  QA  agency  to  lower  certain 
standards  or  to  make  others  more  stringent.     If,    for  example,  no 
provider  is  even  in  minimal  compliance  with  a  particular  stan- 
dard,  the  QA  agency  may  wish  to  lower  its  expectations  with 
respect  to  that  standard.     Conversely,   if  all  or  nearly  all 
providers  are  in  substantial  or  full  compliance  with  a  par- 
ticular standard,   the  QA  agency  may  wish  to  make  that  standard 
more  stringent  so  as  to  improve  standards  of  performance  where 
it  is  judged  that  such  improvement  will  contribute  to  better 
client  outcomes  at  a  feasible  cost.     Alternatively,   the  QA 
agency  may  elect  to  simply  eliminate  the  standard,  especially 
if  the  agency  suspects  that  providers  will  continue  at 
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substantial  or  full  levels  of  compliance  even  in  the  absence 

of  formal  quality  assessment  with  respect  to  that  standard. 
Applicability 

In  assessing  provider  performance  with  respect  to  each 
standard,   the  QA  agency  should  also  make  allowance  for  a  category 
labelled  "not  applicable."     Although,   the  QA  agency  will  have 
determined  during  the  design  phase  which  standards  are  applicable 
to  which  provider  types,   the  agency  may  discover  while  conducting 
assessments  that  certain  standards  originally  judged  applicable 
to  a  particular  provider  type  generally  proved  not  applicable. 
Information  of  this  nature  might  lead  the  agency  to  eliminate 
some  standards  from  the  assessment  of  certain  categories  of 
providers . 

B.     Measures  of  Effectiveness 

The  primary  measure  of  the  effectiveness  of  the  quality 
assurance  mechanism  is  observed  improvement  in  provider  perform- 
ance  (versus  the  standards)   over  time.     Another  measure  of  the  ■■ 
effectiveness  is  the  balance  of  coverage  afforded  to  both 
service  providers  and  service  recipients. 

Three  measures  of  the  effectiveness  of  the  qualitv  assur- 
ance mechanism  with  respect  to  provider  performance  are 
suggested : 

1)  standard  by  standard,   the  change  over  time  in 
the  degree  of  provider  compliance; 

2)  provider  by  provider,   comparative  performance 
on  successive  reviews; 

3)  cycle  by  cycle,  changes  in  the  distribution  of 
the  summary  judgments  rendered. 
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Degree  of  Provider  Compliance,   Standard  by  Standard.     By  the 
time  the  QA  agency  has  completed  three  annual  cycles  of  quality 
assessments    (if  not  sooner) ,   the  agency  will  have  developed  a 
baseline  distribution  of  the  proportion  of  providers  in  full 
compliance,   substantial  compliance,  minimal  compliance  and 
non-compliance  with  each  individual  standard.     Once  this  base- 
line has  been  established,   the  agency  should  monitor  provider 
performance  with  respect  to  each  particular  standard.     If  the 
quality  assurance  mechanism  has  been  effective,  provider  per- 
formance should  have  improved  over  time.     Evidence  of  improve- 
ment would  be  a  lesser  proportion  of  providers  at  the  non- 
compliance/minimal  compliance  end  of  the  spectrum  and  a  greater 
proportion  of  providers  at  the  substantial/full  compliance  end 
of  the  spectrum   (assuming  both  the  standard  and  the  associated 
levels  of  compliance  have  remained  unchanged) . 
Comparative  Performance  On  Successive  Reviews,   by  Provider. 
Another  measure  of  effectiveness  is  the  degree  to  which  a 
particular  provider  demonstrated  increased  compliance  with 
the  standards  on  two  successive  reviews.     One  approach  would 
be  to  analyze  by  priority  class  the  proportion  of  standards 
with  which  the  provider  was  respectively  in  full  compliance, 
substantial  compliance,  minimal  compliance  and  non-compliance. 
If,   for  example,  a  provider  was  in  full  compliance  with  only 
25%  of  all  class  1  standards  upon  first  review,  but  was  in 
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full  compliance  with  75%  of  such  standards  during  the  next 
quality  assessment,   the  effectiveness  of  the  quality  assur- 
ance mechanism  with  respect  to  that  provider  could  be  inferred 
Another  approach  would  be  to  compare  the  successive  summary 
judgments  rendered  each  provider  to  determine  the  likeli- 
hood that  a  conditional  judgment  was  followed  by  a  positive 
judgment,   or  the  likelihood  that  a  negative  judgment  was 
followed  by  either  a  conditional  or  a  positive  judgment. 
Either  analysis  would  assist  the  QA  agency  to  determine  the 
adequacy  of  provider  plans  of  correction  and  the  degree 
to  which  such  plans  had  been  successfully  implemented. 
Distribution  of  Summary  Judgments,   Cycle  by  Cycle.     A  more 
aggregate  measure  of  effectiveness  would  be  the  change  over 
time  in  the  proportion  of  providers  rendered  positive,  con- 
ditional and  negative  summary  judgments.     An  increase  in  the 
number  of  positive  judgments  and/or  a  decrease  in  the  number 
of  negative  judgments  would  be  indicative  that  the  quality 
assurance  mechanism  was  having  the  desired  end  result — im- 
proving the  quality  of  services.     Of  particular  interest 
would  be  the  proportion  of  providers  rendered  a  conditional 
judgment  in  a  prior  review  cycle  who  were  rendered  positive, 
conditional  or  negative  judgments  in  the  most  recent  review 
cycle.     Adequate  allowance  would  need  to  be  made,  however, 
for  variations  in  the  set  of  providers  reviewed  and  for 
changes  made  to  the  standards  or  to  the  decision  rules. 
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Coverage  as  a  Measure  of  Effectiveness 

While  it  is  unnecessary  to  include  all  service  providers 
in  any  single  review  cycle,   the  QA  agency  should  strive  to 
attain  adequate  coverage  of  each  provider  type  and  each  subset 
of  developmentally  disabled  persons   (as  defined  by  age,  develop- 
mental level,  place  of  residence...).     The  greater  the  coverage, 
the  greater  the  likelihood  that  any  observed  positive  effects 
are  in  fact  representative  of  systemwide  gains. 

The  QA  agency  should  thus  monitor  by  provider  type  the  pro- 
portion of  providers  reviewed   (a)    in  the  most  recent  review 
cycle,    (b)    at  least  once  within  the  last  two  review  cycles,  and 
(c)    at  least  once  within  the  last  three  review  cycles.  Like- 
wise,  the  QA  agency  should  monitor  the  number  of  persons  in 
selected  population  subgroups  who  were  covered   (by  virtue  of 
one  or  more  of  their  primary  service  providers  having  been 
included)    in  each  successive  review  cycle.     The  agency  may 
thus  determine  that  an  excessively  high  level  of  effort  is 
being  applied  to  one  provider  type  or  one  population  subgroup 
at  the  expense  of  others,   or  that  one  provider  type  or  popula- 
tion subgroup  has  been  comparatively  ignored  in  the  quality 
assessment  proceedings.     In  either  case,   the  agency  could  modify 
its  procedure  of  selecting  providers  to  correct  such  imbalances. 
C.     Measures  of  Efficiency 

Measures  of  efficiency  should  focus  primarily  on  the 
manpower  and  financial  expenditures  associated  with  the  imple- 
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mentation  of  the  quality  assurance  process.     The  primary  con- 
cern is  whether  the  process  can  be  carried  out  equally  well 
using  fewer  or  less  expensive  resources.     A  related  concern 
is  whether  or  not  the  time  required  to  complete  a  review  is 
so  lengthy  that  it  renders  the  assessment  findings  and 
summary  judgment  cut  of  date  soon  after  their  publication. 
The  factors  to  be  considered  in  evaluating  the  efficiency  of 
the  quality  assurance  mechanism  include  the  number  of  pro- 
fessional person  hours  expended  by  the  QA  agency  staff  and 
consultants  during  a  given  review  cycle,   the  annual  expen- 
ditures by  the  QA  agency  for  quality  assurance  purposes,  and 
the  elapsed  time  to  carry  out  a  quality  assessment  review  of 
a  single  provider. 
Level  of  Effort 

In  order  to  address  this  factor,   the  QA  agency  should 
keep  accurate  records  on  the  number  of  professional  person 
hours  expended  by  agency  staff  and  consultants  for  all  acti- 
vities throughout  the  year.     Thus,   the  QA  agency  should  main- 
tain individual  time  records  which  show  for  each  staff  member 
the  number  of  hours  spent  weekly  in  each  of  the  following 
activities : 

•  provider  assessments 

•  data  assembly  and  data  maintenance  activities  not 
associated     with  any  particular  assessment 

•  training  or  continuing  staff  education 
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•  other  administrative  activities 

•  self-evaluation  activities 

Each  year,   the  QA  agency  should  analyze  the  number  of  hours 
spent  in  each  activity  to  determine  whether  any  of  these 
activities  have  been  either  under  or  over-emphasized. 

Furthermore,   the  QA  agency  should  determine,   by  provider 
type  and  size,   the  average  number  of  hours  spent  in  the 
quality  assessment  review  of  a  single  provider.     As  these 
data  become  available,    the  QA  agency  will  better  be  able  to 
allocate  manpower  to  quality  assessment  activities  during 
future  review  cycles  and  may  be  able  to  identify  certain 
provider  groups  where  either  more  or  less  effort  is  required. 
Comparison  of  data  with  other  states  may  prove  helpful  in  the 
latter  analysis. 
Cost 

The  overall  cost  of  the  quality  assurance  mechanism  is 

- 

not  an  appropriate  measure  of  efficiency  without  regard  to 
the  benefits  dervied.     Suppose,   for  example,   that  the  quality 
assurance  mechanism  in  one  state  costs  twice  as  much  to 
administer   (on  an  annual  basis)   as  a  mechanism  in  another  state. 
If  the  benefits  derived  from  quality  assurance  activities  in  the 
first  state  were  not  substantially  higher  than  the  benefits 
derived  in  the  second,  one  might  prefer  the  second  state's 
procedures.     On  the  other  hand,  if  the  benefits  in  the  first 
state  were,   for  example,  more  than  twice  as  great  as  the 
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benefits  in  the  second,   the  first  state's  approach  is  likely 
to  be  preferred  despite  the  higher  cost. 

It  is  suggested  above  that  the  benefits    (effectiveness)  of 
a  quality  assurance  mechanism  can  best  be  measured  in  terms  of 
changes  in  provider  performance   (versus  the  standards)  over 
time.     As  was  pointed  out,   however,   there  are  several  ways  of 
measuring  changes  in  provider  performance  and  thus  it  is  doubt- 
ful that  a  single  measure  of  cost-effectiveness  can  be  developed 
at  this  time.     Nor  is  the  QA  agency  likely  to  be  able  to  attri- 
bute a  cost  to  each  single  review,   or  even  to  all  reviews  of  a 
given  provider  type.     Thus,    it  is  suggested  that  the  most  useful 
measure  of  cost  is  the  relative  financial  expenditure  on  quality 
assurance  as  compared  to  service  delivery. 

Put  in  simplest  terms,   persons  with  developmental 

disabilities  are  served  in  three  categories  of  programs: 

c     programs  designed  for  persons  with  developmental 
disabilities    (e.g.   residential  or  day  programs  for 
the  mentally  retarded) ; 

e     programs  designed  for  special  needs  populations, 
in  which  persons  with  developmental  disabilities 
are  a  sub-population,   usually  a  minority  (e.g. 
vocational  rehabilitation) ; 

•     generic  programs  serving  the  population  at  large. 

Because  persons  with  developmental  disabilities  are  a 

small  minority  in  most  programs  in  the  latter  two  categories, 

it  may  be  very  difficult  to  accurately  estimate  total  annual 

expenditures  for  service  delivery.     Conversely,   it  should  be 

possible  to  estimate  with  reasonable  accuracy  the  amount  of 
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public  funds  expended  annually  within  the  state  for  the  provi- 
sion of  services  designed  for  persons  with  developmental  dis- 
abilities.     If  this  estimate  is  available,   the  QA  agency  can 
at  least  compare  the  annual  expenditures  for  quality  assurance 
in  its  state  to  the  annual  public  expenditures  for  the  delivery 
of  such  services.     Although  no  general  standard  will  apply,  states 
may  wish  to  establish  a  policy  that  expenditures  for  quality 
assurance  should  range  between   (say)    two  percent  and  four  per- 
cent of  the  annual  expenditures  for  such  service  delivery. 
Elapsed  Time 

The  QA  agency  should  maintain  records  which  identify  the 
start  date  and  end  date  of  each  major  activity  within  a  service 
provider  assessment.     In  particular,   agency  records  should 
indicate  the  date  upon  which  the  provider  was  notified  of  the 
impending  assessment,   the  date  upon  which  data  assembly  began, 
the  dates  on  which  the  site  visit  was  conducted,   and  the  date 
upon  which  the  summary  judgment  and  assessment  findings  were 
transmitted  to  the  service  provider.     The  primary  measure  of 
efficiency  from  the  standpoint  of  elapsed  time  between  the 
notification  of  the  provider   (subtask  5.3)   to  the  communication 
of  the  summary  judgment  and  supporting  documentation 
(subtask  9.1).     More  detailed  analyses  of  the  elapsed  time  to 
complete  each  of  the  intervening  steps  may  also  be  of  use  to  the 
agency  in  streamlining  its  quality  assurance  procedures  in 
future  review  cycles. 
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D,     Measures  of  Equity 


Concerns  regarding  the  equity  of  a  quality  assurance 

mechanism  include  the  assurance  of  due  process  and  impartiality, 

and  the  absence  of  any  infringement  upon  the  rights  of  persons 

with  developmental  disabilities  or  upon  the  rights  of  service 

providers.     Several  design  elements  of  the  model  quality 

assurance  mechanism  were  specifically  included  to  assure  due 

process  and  impartiality.     The  following  are  of  particular 

importance  in  this  regard: 

©     all  standards  should  be  in  writing  and  should 
be  made  available  in  advance  to  all  service 
providers; 

©  the  priority  class  of  each  standard  and  the 
several  levels  of  compliance  should  also  be 
identified; 

©     the  QA  agency  should  have  established  unbiased 
and  written  decision  rules  for  selecting  service 
providers  for  review,   including  decision  rules 
for  establishing  reaudit  requirements; 

©     the  QA  agency  should  have  established  written 
procedures  for  judging  the  accuracy  of  data 
supplied  by  sources  external  to  the  QA  agency; 
such  sources  include  both  the  service  provider 
and  secondary  sources; 

©     the  QA  agency  should  have  established  and 

followed  procedures  for  training  and  maintaining 
the  competence  of  persons  employed  as  quality 
assessors; 

•     the  QA  agency  should  have  established  written 
procedures  for  protecting  client  rights  by 
limiting  access  to  data,  maintaining  confidentiality 
of  clients  and  allowing  clients  to  refuse  participa- 
tion in  data  gathering  activities; 
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©     the  QA  agency  should  have  established  procedures 

for  allowing  other  interested  parties  to  contribute 
to  the  quality  assessment  process  prior  to  the 
rendering  of  the  summary  judgment; 

©  the  QA  agency  should  have  established  unbiased  and 
written  decisions  rules  for  reaching  summary  judg- 
ments and  for  imposing  sanctions; 

©     the  QA  agency  should  have  established  and  followed 
written  procedures  for  deciding  whether  or  not 
to  publicize  summary  judgments,   sanctions  or 
recommendations  for  corrective  action. 

If  all  of  the  above  steps  have  been  carried  out,   equity  con- 
cerns focus  on  two  questions:     Were  the  written  procedures 
and  decision  rules  appropriate?     Were  they  followed?  The 
most  useful  approach  to  answering  these  questions  is  for  the 
QA  agency  to  analyze  the  nature  of  all  appeals  and  challenges 
regarding  its  activities  and  judgments. 

The  QA  agency  should  thus  keep  records  of  the  number  of 
summary  judgments  appealed  and  the  proportion  of  such  appeals 
upheld.     The  source  of  the  appeal    (the  service  provider,  per- 
sons with  developmental  disabilities,  members  of  their  family, 
their  guardians  or  advocates,  or  other  interested  parties) 
should  also  be  noted  as  should  the  specific  reason  for  the 
appeal.     An  analysis  of  the  frequency,   the  source  and  the  reason 
for  all  appeals,  especially  those  appeals  upheld,  will  suggest 
to  the  QA  agency  how  its  quality  assurance  mechanism  might  be 
modified . 

During  each  service  provider  review,  the  QA  agency  should 
also  maintain  records  of  all  written  challenges  and  all  oral 
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complaints  regarding  the  quality  assurance  proceedings.  Such 
challenges  and  complaints  are  likely  to  focus  upon  the  alleged 
inappropr iateness  of  individual  standards,   the  degree  of  inter- 
ference with  routine  provider  activities,   the  invasion  of  privacy 
of  persons  with  developmental  disabilities,   or  the  inappropriate- 
ness  of  selected  assessment  findings.     A  certain  portion  of  the 
self  evaluation  activities  should  focus  on  an  analysis  of  these 
challenges  and  complaints  to  determine  whether  any  changes  in 
the  quality  assurance  process  are  warranted. 
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